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Preface 


At lunch on March 25, 1980, at a workshop on Artificial 
Intelligence and Philosophy at Stanford's Center for Ad- 
vanced Study in the Behavioral Sciences, one of the par- 
ticipants suggested that we would all have been better off 
if Kant had written his Critigwe of Pure Reason in Lisp. 
Looking up in alarm, Ilocked eyes with an egually shocked 
participant, in the person of John Haugeland. So began a 
fast friendship and intellectual partnership that lasted until 
May 22, 2010, when John tragically succumbed to a heart 
attack at his own Festschrift at the University of Chicago.” 

Haugeland was a philosopher, trained in existentialism 
and the philosophy of Heidegger, who devoted consid- 
erable resources to exploring the philosophical founda- 
tions of artificial intelligence (A1). I met him just as I was 
completing a doctorate in computer science at the Artifi- 
cial Intelligence Laboratory at MIT, From that technical 
background I have spent much of my career reaching in 
the opposite direction from John, but toward the same ter- 
ritory: the philosophical imbrications of computation and 
mind. Our backgrounds were contrapuntal, but the com- 
mon interest arose out of a deeper similarity: Haugeland's 


1. İ wrote my dissertation on Lisp; it was not for lack of familiarity 
with the programming language that I was appalled. 

2. Haugeland lived for another month, passing away on June 23, but 
never regained consciousness after the conference, 


father was a machinist;” my own, a theologian. Neither of 
us had in fact ever been purely one thing or the other. We 
grew up and dwelt in the same textured landscape between 
the technical and the philosophical—a landscape we were 
destined to spend hundreds of hours exploring together. 
“Ihats what I was just about to think!” became a staple of 
our endless conversations, 

In the circumstance, I was unable to give a paper at 
Haugeland's Festschrift. Ihis work is an attempt to lay out 
some of the views I would like to have presented. 


Ihanks especially to Güven Güzeldere, Amogha Sahu, 
and Henry Ihompson for reading multiple drafts, and for 
extended ensuing discussions of these topics. Thanks as 
well to Christoph Becker, Jim Brink, Tony Chemero, Ron 
Chrisley, Nils Dahlbâck, Sandra Danilovic, Gillian Ein- 
stein, Lana el Sanyoura, Robert Gibbs, Vinod Goel, Stefan 
Heck, Steven Hockema, Atoosa Kasirzadeh, Jun Luo, Jim 
Mahoney, Tobias Rees, Murray Shanahan, Arnold Smith, 
and Rasmus Winther—as well as two anonymous review- 
ers. I also want to thank Ron Chrisley and the members 
of the “Minds, Machines, and Metaphysics” community 
(3M), first convened at the University of Indiana in the 
19905, that Ron has recently reconstituted, online, bring- 
ing together many of students with whom it has been my 
honor to work over the years. I know for a fact that no one 
mentioned here agrees with everything in this book, but I 
am nevertheless extremely grateful to them all. I can assure 


3. While a graduate student, Haugeland built himself a printer by 
assembling a grid of relays, to which he attached bent paper clips that 
reached up and pulled down the keys on a scavenged IBM Selectric 
iypewriter. He fabricated appropriate electronics and wrote a pro- 
gram to control the device, which served him for many years. 


each and every one that the book is immensely better for 
their interventions. 

'Ihanks to Robert Prior, Executive Editor at the MIT 
Press, for stewardship of this book and for faith and sup- 
port over many years, and to Elizabeth Agresta for assis- 
tance in publication details. 

I am also extremely grateful to Reid Hoffman for sup- 
porting the writing of this work, the research on which it is 
based, and the chance to pursue these topics over coming 
years. I also want to acknowledge his challenge to me, on 
December 16, 2017, to think about how we humans should 
approach living with synthetic forms of intelligence. 1 view 
this book as a chance to give voice to some initial thoughts. 

Ihanks beyond words to Gillian Einstein, as well— 
without whom 1 dare not imagine. 


Introduction 


Meicher deep learning, nor other forms of second-wave Al, 
nor any proposals yet advanced for third-wave, will lead 
to genuine intelligence. Systems currentiy being imagined 
will achieve formidable reckoning prowess, but human- 
level intelligence and judgment, honed over millennia, is of 
a different order. It reguires'getting up out” of internal rep- 
resentations and being committed to the world as world, in 
all its unutterable richness. Only with existential commit- 
ment, genuine stakes, and passionate resolve to hold things 
accountable to being in the world can a system (human or 
machine) genuinely refer to an object, assess ontological 
schemes, distinguish truth from falsity, respond appropri- 
ately to context, and shoulder responsibility. 

Does that doom A1? No. Automated reckoning systems 
will transform human existence. But to understand their 
capacities, liabilities, impacts, and ethics, and to under- 
stand what assemblages of people and machines should be 
assigned what kinds of task, we need to understand what 
intelligence is, what Al has accomplished, and what kinds 
of work reguire what kinds of capacity. Only with a care- 
fully delineated map can we wisely choreograph the world 
we are developing—the world we will jointly inhabit. 

'Ihis book is intended as a contribution to that carto- 
graphic project. It is written out of a belief that the rise of 
computing and Al is of epochal significance, likely to be as 
conseguential as the Scientific Revolution—an upheaval 


that will profoundiy alter our understanding of the world, 
ourselves, and our (and our Als) place in that world. So 
encompassing is the reconfıguration being catalyzed by 
computing and Al that we need fundamentally new on- 
tological and epistemic frameworks to come to terms with 
it—new resources with which to ask such ultimate gues- 
tions as who we are, who to be, what to stand for, how to live, 

With an eye toward such guestions, this book develops 
some intellectual tools with which to assess current devel- 
opments in Al—especially recent advances in deep learn- 
ing and second-wave Al that have led to such excitement, 
anxiety, and debate. Ihe book will not assess specific proj- 
ects, or recommend incremental advances. Instead, I will 
adopt a general strategy, in pursuit of two aims: 


1. To unpack the notion of intelligence itself, in order 
to understand what sort(s) we humans have, what 
sort(s) Al aims at, what Al has accomplished so far, 
what can be expected in the foreseeable future, and 
what sorts of tasks can responsibly be assigned to 
systems currentİy constructed or imagined. 

2. To develop a better understanding of the underly- 
ing nature of the world, to which 1 believe all forms 
of intelligence are ultimately accountable. 


Ihe second concern gives the book a strongly ontological 
flavor. In the end, I argue (i) that the deepest reason for 
the failure of fırst-wave Al was the untenable ontological 
world view on which it was founded, (ii) that the most im- 
portant insight of second-wave Al is the window it gives 
us onto an alternative ontological perspective, and (ii) that 
the nature of reality implies that constructing anything 
warranting the label “artificial general intelligence” (AGI) 
will teguire developments far beyond and guite different 


from those imagined in either first- or second-wave AL in- 
volving various forms of committed, participatory engage- 
ment with the world. 


I use judgment for the normative ideal to which I argue 
we should hold full-blooded human intellişence—a form 
of dispassionate” deliberative thought, grounded in ethi- 
cal commitment and responsible action, appropriate to the 
situation in which it is deployed. Not evety human cogni- 
tive act meets this ideal—not even every conscious act of 
every person we call intelligent. 1 claim only that jadgment 
is the standard to which human thinking must ultimately 
aspire.” 

Judgment of this sort, I believe, is a capacity we strive 
to instill in our children, and a principle to which we hold 
adults accountable, It is an achievement that far tran- 
scends individuals—a resource that has been forged, over 


1. Or any that have been proposed for third-wave Al, 
2.“Dispassionate” (and'disinterested” when 1 use the term) in its orig- 
inal sense of being fair, unbiased, open-minded, and free of prejudice. 
I neither mean nor intend to suggest that jadgment should (or can) 
lack in care or commitment. On the contrary, as argued in chapter 11 
and again in chapter 13, judgment must be simultaneously passionate, 
dispassionate, and compassionate. 
3. Ihis is not to suggest that judgment, as a regulative ideal, is remote 
from either consciousness or experience. One of the most important 
accomplishments of the historical development of culture, I believe, 
in diverse forms around the world, is to have established standards of 
judgment as a background condition on what it is to be a responsible 
adult. Ihe fact that contemporary rending of the fabric of public dis- 
course (perhaps abetted by digital technologies) is so widely decried 
stands witness to the fact that such norms have not been forgotten, 
even if they appear to be under threat. 

Coming to understand what it is or would be to hold the human 
condition accountable to such an ideal is an added benefit of docu- 
menting what Al systems are, and what they are not. 


thousands of years and in diverse cultutes,* as a foundation 
for rationality, thought, and deliberative action, into which 
individuals must be recruited. It need not be articulate, 'ra- 
tionalistic,” or independent of creativity, compassion, and 
generosity —failings of which (especially formal) logic is 
often accused. Rather, by jadgment I mean something like 
what we get at (or should be getting at) when we say that 
someone “has good judgment”: a form of thinking that is 
reliable, just,” and committed—to truth, to the world as 
It is, 

With judgment in view as the ultimate goal of general 
intelligence, I examine the history of artificial intelligence, 
from its fırst-wave origins in what Haugeland dubbed 
“Good Old Fashioned AT” (GOFAJ) to such contempo- 
rary second-wave approaches as deep learning. My aim is 
neither to promote nor to criticize—but to understand. 
What assumptions underlie the various technologies we 
have constructed? What conceptions of intelligence have 
been targeted at each stage? What kinds of success have 
been achieved so far, and what can be expected in the fu- 
ture? What aspects of judgment will contemporary Al sys- 
tems reach, and what aspects have they not yet begun to 


4. Ihe development of judgment, we can safely presume, was accom- 
plished without reguiring alteration in DNA or neural architecture, 
5. Philosophical readers may balk at the indusion of justice and ethics 
inanorm on truth. As will become increasingiy clear, I take accoynt- 
ability to the world to be something of an “ur-norm” that underlies not 
only truth but ethics, care, and compassion as well. 'To see how and 
why that is true, however, reguires understanding what the “world” is, 
how ontology and truth arise, how existential commitment is a pre- 
condition for ontology, and so on, An argument for this metaphysi- 
cal position is beyond the compass of this book; I will take up that 
project elsewhere, but for an initial glimpse see my On the Origin of 
Objects (Cambridge, MA; MIT Press, 1996). 


approach? And to up the ante, and in order to bring into 
view one of the most important issues facing us today: can 
articulating a conception of judgment provide us with any 
inspiration on how we might use the advent of Al to raise 
the standards on what it is to be human? 

I use the term reckoning for the types of calculative 
prowess at which computer and Al systems already excel — 
skills of extraordinary utility and importance, on which 
there is every reason to suppose computers will continue to 
advance (ultimately far surpassing us in many cases, where 
they do not do so already), but skills embodied in devices 
that lack the ethical commitment, deep contextual aware- 
ness, and ontological sensitivity of judgment. Ihe differ- 
ence between reckoning and judgment, which I will argue 
to be profound, highlights the need for a textured map of 
intelligence's kinds* —a map in terms of which to explain 
why reckoning systems are so astonishingly powerful in 
some respects, yet fall so spectacularly short in others. 


Four caveats. First, the book is not a comparison of hu- 
mans and machines. I see no reason to doubt that it may 
someday be possible to construct synthetic computational 
systems capable of genuine jadgment. Or perhaps eguiva- 
lently: if, as may happen, we construct synthetic creatures 
capable of evolving their own civilizations, or of incre- 
mentally participating in ours, nothing in my argument 


6. Psychology has developed detailed conceptual maps of the human 
psyche, distinguishing such capacities as cognition, sensation, mem- 
ory, and so on. As noted in chapter 2, the conception of intelligence 
on which Al was founded was very general, not making any such 
theoretical distinctions. I believe the kind of map we need in order 
to assess Al, moreover, is of a different order from that provided by 
psychology—in part because computers are opening up vast regions 
unoccupied by humans or nonhuman animals, 


militates against the possibility that, in due course, such 
creatures might themselves evolve judgment—much as we 
have, and perhaps with no more explicit understanding of 
their capacities than we have of our own. Nor am 1 argu- 
ing that cyborgs or other human-machine assemblages, 
for that reason alone, will be challenged in regard to their 
capacity for authentic judgment. My claims are just two: 
(i) the systems we are currently designing and building are 
nowhere near that point; and (ii) no historical or current 
approaches to Al, nor any | see on the horizon, have even 
begun to wrestle with the guestion of what constructing or 
developing judgment would involve, 

Yet attempting to reach this conclusion by drawing a 
distinction between DNA- and silicon-based creatures 
would be a grave mistake, in my view—chauvinist, sen- 
timental, and fatally shallow. Rigor demands that we ar- 
ticulate a space of possible kinds of intelligence in terms 
of which Als, humans, and nonhuman animals can all be 
evenly and nonprejudicialIy assessed. 

Second, as will progressively emerge, jadgment in the 
sense | am defending it is an overarching, systemic capac- 
ity or commitment, involving the whole commitment of a 
whole system to a whole world. 1 do not see it as an isolable 
property of individual capacities; nor do 1 believe it is likely 
to arise from any particular architectural feature—includ- 
ing any architectural feature missing in current designs. 
Readers should not expect to fınd specific architectural 
suggestions here, or recommendations for technical repair. 
Ihe issues are deeper, and the stakes higher, than can be 
reached by any such approach. 

Third, Tam fully aware that the conception of judgment 
I will defend does not fit into any standard division be- 
tween “rational thought, on the one hand, and “emotion” 


or “affect, on the other. On the contrary, one of my aims 
is to unsettle reigning understandings of rationality, in 
part to break it up into different kinds, but also to sug- 
gest that reason in its fullest sense—reason of any sort to 
which we should aspire—necessarily incorporates some of 
the commitments and compulsions to action that are often 
associated with affectual states. Ihese moves arise out of 
a İarger commitment: if we are to give the prospect of Al 
the importance it deserves, we must not assume that time- 
honored conceptions of rationality will survive unscathed. 
Fourth, although I take seriously many of the critigues 
of fırst-wave Al articulated in the 1970, this book is by no 
means intended to be an updated treatise along the lines 
of Dreyfuss What Computers Cant Do.” On the contrary, 
one of my goals is to develop conceptual resources in terms 
of which to understand what computers can do. In fact, 
the entire discussion is intended to be positive. I do not 
plead for halting the development of AJ, or argue that we 
should bar Al systems from functioning in situations of 
moral gravity. (When landing in San Francisco, I am glad 
the airplane is guided by sophisticated computer systems, 
rather than by pilots peering out through the fog looking 
for the airport.) I am also not wotried, at least here, about 
whether Al systems will grow more powetful than we hu- 
mans, or that they will develop their own consciousness. 
Andi take seriously the fact that we will soon need to learn 
how to live in productive communion with synthetic intel- 
ligent creatures of our own (and ultimately their) design. 
wo things do terrify me, though: (i) that we will rely 
on reckoning systems in situations that reguire genuine 


judgment; and (ii) that, by being unduly impressed by 


7. Hubert Dreyfus, What Computers Can't Do: A Critigue of Artificial 
Reason (New York: Harper & Row, 1972). 


reckoning prowess, we will shift our expectations on hu- 
man mental activity in a reckoning direction. Current 
events give me pause in both respects. The calls to which I 
believe we should respond, and to which I hope this book 
will draw our attention, are: (i) that we learn how to use Al 
systems to shoulder the reckoning tasks at which they ex- 
cel, and not for other tasks beyond their capacity; and (ii) 
that we strengthen, rather than weaken, our commitment 
to judgment, dispassion, ethics, and the world. 


1 — Background 


There was excitement the first time too. In the 1960s and 
19705 artifıcial intelligence was in its first Hush. Comput- 
ers stunning power had only recently come into focus, 
electrifying the newly minted AT labs. What gripped our 
imagination was not just computings potential impact on 
society, impressive as that was, but the thought that we 
humans, too, might be computers. The idea unleashed vi- 
sions of grandeur. If we could just “program things up,” we 
dreamed, we could put paid to thousands of years of phi- 
losophy, surround ourselves with intellişent companions, 
and understand the human condition. 

It is easy to disparage that bravado now. Ihose “first- 
wave Al” systems were brittle, unconducive to learning, de- 
feated by uncertainty, and unable to cope with the world's 
rough and tumble. Yet there were also successes, many of 
which paved the way for what is now commonplace—on- 
line commerce, navigation systems, social media. But in a 
long-standing pattern, once tasks were automated they no 
longer seemed to reguire so much intelligence. Gradually 
che ebullience waned. 

here is excitement once again—for “second-wave” Al, 
this time, for a new “Al spring. Deep learning and afhli- 
ated statistical methods, backed by unimagined computa- 
tional power and reams of Big Data, are achieving levels of 
performance that trounce the achievements of the earlier 
phase. Ihe profession is psyched, the commercial world 


scrambling, the press giddy with anticipation. But there 
is also anxiety. What captures our imagination is not s0 
much awe, this time, as conseguence. Exhilaration is being 
replaced by terror—that new and perhaps alien Als are go- 
ing to take over our jobs, our lives, our world. 

Are we genuinely on the cusp of what we only prema- 
turely imagined in the 1970s? Or is this, too, a phase des- 
tined to pass? One thing is sure: there is no dearth of opin- 
ions, lestimonials about second-wave Al Hood the web, 
along with comparisons with fırst-wave, and, as new limi- 
tations come into view, proposals for “third-wave” Al—for 
what it will take to achieve the ultimate goal of “artificial 
general intellişence” (AGI). There has certainly been as- 
tounding progress. World-beating Go programs, service- 
able machine translation, and uncanny image recognition 
are genuinely stunning. Respected scientists are serious 
about the magnitude of the impending upheaval—the 
prophets of doom as well as the inveterate triumphalists, 

But respected scientists were serious 50 years ago, t00. 


To take stock of what is happening, we need to understand 
the ontological, epistemological, and existential assump- 
tions on which each wave of Al has been based. Viewing 
things historicaliy will help us judge the capacities, impli- 
cations, and limitations of the fırst two stages, and assess 
prospects for a third. It will also allow for a more discerning 
analysis than the standard characterizations of fırst-wave 
Al as based on “handcrafted symbolic representations” 
and second-wave on “statistical pattern matching over large 
data sets” Taking the measure of each, and soberly assaying 
the future, reguires something deeper: not just a unifying 
architectural perspective that subsumes both options (and 
perhaps others as well), but, more broadIy, an understand- 


ing of what intelligence is, on the one hand, and what the 
world is like, on the other, It is time to channel McCull- 
och: what is the world, such that we can understand it— and 
who are we, such that we can understand the world? 

Ihree methodological preliminaries. First, as already 
indicated, it is imperative not to frame the debate in terms 
of human and machine. The problem is not just that those 
labels are vast, vague, and emotionally charged. More seri- 
ously, to avoid ideology and prejudice, we need indepen- 
dent evaluative criteria, To judge what kind(s) of intelli- 
gence cutrent Als have, what kind(s) we humans have, and 
what kinds are on offer for both machines and people, we 
need an understanding of “intelligence” and its “kinds” not 
circularly defined in terms of the entities to which we want 
to apply it. 

Second, it is important to deflect potential confusion 
due to terminological differences between computer sci- 
ence and adjacent frelds. A variety of classic terms from 
logic, philosophy, linguistics, and epistemology, having to 
do with relation of an intentional (representational, mean- 
ingful) system to the wider world, have been repurposed 
in computational parlance to refer to causal behaviors and 
arrangements within the confines of the machine itself, 
Especially important are meaning, semantics, reference, and 
interpretation. Consider the phrase “the semantics of pro- 
gram P” Many (including 1) would take this term to refer 
to a relation between P and the world or task domain that 
P is'about” (that its data structures represent, in which it is 
deployed, about which it performs calculations). But from 
a perspective I call blanket mechanism, which has largely 


1. Warren McCulloch, “What is a Number, that a Man May Know 
Te, and a Man, that He May Know a Number?, General Semantics 
Bulletin, no. 26/27 (1960): 7—18. 


r 


engulfed computer science, the phrase is used to refer to 
the behavioral conseguences, within the computer system, 
of P's being executed. In computer science, that is, the term 
“semantics” no İlonger takes one outside the boundaries of 
a mechanistic device. Since my concerns in this book are 
with the very “into the real world” relations that the terms 
traditionally referred to, I will use them here in their classic 
sense, marking those points where confusion is most likely 
to arise.” 

Ihird, the gravity of the stakes suggests addressing the 
issues ata relatively high level of generality. Some guestions 
are foundational: What is intelligence? What are its physi- 
cal limits? What lies beyond the line of possibility, so that 
we can give up longing for it? And on this side: Where are 
we, now, and where are our creations? How will we want— 
how should we want—to live with other forms of intel- 
ligence, including not just products of natural evolution, 


2. Not a few computer scientists have suggested, over the years, that 
I'get with the program” and realize that these internal mechanical 
relations and behaviors are what the terms semantics, reference, inter- 
pretation, and the like now mean. But the suggestion raises numerous 
difficulties. First, 1 need terms that mean what these terms classicalIly 
meant. If I were to accept the redefinitions, or (more plausibly) were 
to avoid them in order not to sow confusion, I would lose any way 
to talk about what I do want to address. Second, I believe that re- 
taining the classical meanings is necessary in order to understand the 
relations among Al, computing, mind, language, and thought. Third, 
even scientificaliy, I believe we need to retain the classical meanings in 
order to get at technical issues within computing itself (see my Com- 
putational Reflections, forthcoming). Fourth, and most importantiy, 1 
have no interest in supporting blanket mechanism in general, which 
I see infecting wider and wider swaths of intellectual inguiry. On the 
contrary, I believe it is vital that we not submit to the rising tide of en- 
tirely mechanistic world views and explanations. So the redefınitions 
and repurposings are moves | want to resist. 


but entities of our own devising, and eventually of theirs 
as well? What impact will and should the development of 
synthetic” intelligence have on human intelligence, on our 
sense of self, on standards for humanity? 

There are also practical guestions. What sorts of intel- 
ligence are reguired to perform different sorts of task— 
driving a car in the city, reading an X-ray, building houses, 
teaching children, detecting racism, routing propaganda 
and fake news? How can we responsibly assign specific 
kinds of work to appropriate confıgurations of people, ma- 
chines, and processes? What divisions of labor, between 
and among organisms and devices and communities and 
systems and governments, are sustainable, honorable— 
and humane? 

'Ihese guestions target the future. To address them, we 
need a grip on the past. 


3.A better word than “artificial/ if it can ever be achieved. 


2 — History 


İt is not random that classical fırst-wave Al was based on 
symbolic representation. Its approach, immortalized in 
Haugelandis indelible phrase “good old-fashioned Artifı- 
cal Intelligence” (GOFAT),' grew out of four vaguely Car- 


tesian assumptions: 


Cı The essence of intelligence is thought, meaning 
roughly rational deliberation. 

C> The ideal model of thought is logical inference 
(based on “dear and distinct” concepts, of the sort 
we associate with discrete words). 

C3 Perception is at a lower level than thought, and 
will not be that conceptualiy demanding.” 

C4 The ontology of the world is what I will call 
formal? discrete, well-defined, mesoscale objects 


1. John Haugeland, Artificial Mmtelligence: The Very Idea (Cambridge, 
MA: MIT Press, 1985), 112. 

2, It is not that anyone believed that perception would be simple. 
Rather, the conceit was that perception, perhaps because so well ex- 
emplifed in nonhuman animals, should not only be viewed as lower- 
level than real intelligence, but also, and crucialIy, much less mysteri- 
ous—not of the same order as the really profound guestion of what 
genuine intellişence was, how it had emerged, how thought was pos- 
sible in a mere physical device, etc. (That we are perceiving creatures 
was never thought to be an argument for dualism, or a reason to 
doubt our material constitution.) 

3. Ihis is by no means a standard definition of formality. Ihe 
term is used in a variety of ways, to mean syntactic, abstract, and 


exemplifying properties and standing in unambigu- 
ous relations, 


Regarding Cı, when Al was founded, the term "intelligence" 
was given no very specific meaning. İhe aim was simply to 
construct a machine that intuitively seemed “smart” —that 
could think, emulate human tationality, or demonstrate 
cognitive prowess. Since that time, people have substan- 
tially enlarged the scope of mental activity toward which 
they direct AL—most obviously by induding perception, 
action, and categorization within mainstream Al research, 
but also by exploring emotion, cognitive development, the 
nature of assertion and denial, and so forth, But in the 
beginning, conceptual representation was largely taken as 
given, and the focus was directed toward a roughly logicist 
conception of thinking and reasoning. 


There is nothing specifically computational about assump- 
tions Cı—C4.* An additional insight was reguired in order 
to tie them into computation, arising from nineteenth- and 
twentieth-century developments in logic in the work of 
Boole, Peirce, Frege, and others. It bears a word of expli- 
cation here, partly because appreciation of it has waned,” 
but also because it allows us to frame the challenge that all 
intelligence must face. 


mathematical, among other things. See page 324 of On the Origin 
of Objects for an analysis, along the lines suggested here, of the com- 
mon thread underlying all of these readings— of formality, ay fond, as 
“discreteness run amok.” 

4. If anything, the association of computing with this model of intel- 
ligence ran the other way: Turing's proposal for his eponymous ma- 
chines was based on his logicist conception of intelligence. 

5. Ihough contemporary programmers may not recognize the formu- 
lation spelled out on the following pages, Turing would have found it 
unexceptional, 


Ihe insight rests on a fundamental thesis about repre- 
sentation and the nature of physical causality. It takes the 
form of a claim that it is possible to build a system with 
four crucial properties, which together allow (us to take) it 
to reason” or “process information”: 


Pı he system works, mechanically, in ways explica- 
ble by science. We can build such devices. Nothing 
spooky is reguired. No magic or divine “inspira- 
ton”; no elixir of life or soul. 

P2 Ihe systems behavior and constituents* support 
semantic interpretation—that is, can be taken to 
be about (mean, represent, etc.) facts and situations 


in the outside world.” 


6. Philosophers draw a distinction between a systems behavior or 
identity as an integrated whole, and its internal constituents (data 
structures, ingredients, etc.) —with properties of the former, in the 
human case, being called personal; the latter, subpersonal. The difter- 
ence is easiest to understand in ethical cases. While we might hold 
you accountable for doing something egregious, if neuroscientists 
were to trace the reason for your misbehavior to abnormal function- 
ing of your brain, it would be odd to say“Bad amygdala!” But the dis- 
tinction between personal and subpersonal is hard to draw carefuliy, 
even in the human case. Are memories, which clearly admit of seman- 
tic interpretation—in fact are comprehensible only in such terms— 
constituents of your mind, or external (personal-level) behaviors? 
'Ihe distinction matters greatly in some cases—including with re- 
spect to the notion of judgment being developed here, which I will 
argue to be an irrevocabiy personal” or"“systemic” capacity, rather than 
being the capacity of any ingredient or subpersonal part. It is not, 
however, relevant to this origin story about Al, 
7. As mentioned in chapter 1, this is not what “semantics” means in 
computer science, where the term has come to be used as a name for 
the mechanicaliy individuated behavioral conseguences of a program 
or data structure, For details see my forthcoming Computational Re- 
Jlections. More generali, the claim that computation is a semantic or 


From these two properties it follows that a wedge opens 
up between two ways of understanding computers: 
(i) what they do, when that is understood “under interpre- 
tation —that is, in terms of the entities in the world that 
the representations signify; and (ii) how tbey work, as (un- 
interpreted) causal mechanisms. Examples of the former 
(what they do under interpretation) include calculating 
the prime factors of a large number, fıguring out the short- 
est route through all American state capitals, predicting 
the outcome of the fall election, telling time. These activi- 
ties are not described in terms of what is going on inside 
the machine, but in terms of what the world is or might be 
like, based on an interpretation of the machines output or 
behavior. Predicting that the government will fall is a pre- 
diction about politics, not (in the first instance) about the 
characteristics of an output or data structure, Information 
that Montreal is north of Ottawa has to do with how the 
world is arranged, not with bits or channels or measures 
or symbols. 

Understanding intelligence under interpretation is 
utterİy commonplace.? If I ask “whats on your mind?” 
you are likely to reply “I'm thinking that M 
where the blank is flled with a description of the world, 
not of your thoughts as uninterpreted causal patternings 
(few are the people who would say “1 have just increased 


intentional phenomenon at all is increasingly challenged (see, e.g, 
Gualtiero Piccinini, Physical Computation: A Mechanistic Account, 
Oxford: Oxford University Press, 2015), though 1 stand by my claim. 
8.“İnterpretation” is another term that is given an internal, mechanist 
reading in computer science, but that I use here in its normal English 
sense, as having to do with the world or intentional subject matter. 

9. İn fact intelligence only is intelligence under interpretation. “Being 
intelligent” is not a description of uninterpreted causal behavior or 
mechanical confıguration. 


my cortisol level by 2.796 and sent 10” neural firings across 
my corpus callosum”).“ 

Principles Pı and P2 are recognized as fundamental to 
the structure of logic, and underlie the “formal symbol ma- 
nipulation” construal of computing as well, While I believe 
the next two also apply to both logic and computing, they 
are less commonly highlighted or remarked. 


P3 The system is mormatively assessed or governed" 
in terms of the semantic interpretation. 


What matters about such systems— whether they are right 
or wrong, true or false, useful or useless— has to do with 
an assessment of what they are interpreted as representing 
or saying, İs it true that if we launch in that direction we 
will ultimately get to Saturn? That there are 78,498 primes 
less than a million? Ihat at this rate it will take 87 years to 
pay off our mortgage? It does not matter how efficient an 
algorithm is if it produces a misleading or wrong answer. 


10. 1 will have more to say about semantic interpretability later. For 
now, it is important merely to note that the issues of whether the in- 
terpretations are arbitrarily attributed and thus changeable at whim 
(as Searle presumes in his Chinese Room example), whether they are 
anchored or grounded in the world (in what is known as the “symbol 
grounding problem”), and whether the semantics is “original” or “de- 
rivative” (or “authentic” and “inauthentic,” as discussed by Searle and 
Haugeland) are at least partially independent. See chapter 7, pages 
78—79. 

ıı. İ would say “normatively evaluated,' except that, like “semantics,” 
the term “evaluate” has come to take on a purely behavioral meaning 
in computer science, at odds with its historically normative sense of 
worth, propriety, being right” or true, etc. 

By'governed” I do not mean causally or mechanistically controlled, 
but something doser to what is meant by saying that a democratic 
society is 'ultimately governed by respect and trust” — or that physical 
events are 'governed by the laws of nature.” 


Ihe fourth point is the most difficult to understand, 
but the most metaphysically basic. It raises a fundamental 
challenge for all forms of intelligence. I would even go so 
far as to say that it is what establishes tbe problem that intel- 
ligence is “designed” to solve. 1 will frame it in terms of effec- 
tiveness, meaning “something that can be done by physi- 
cal or mechanical means” Setting aside complexities due 
to mathematical modeling, this is the sense that computer 
science studies under its label of “effective computability.” 
Subtleties abound, but for present purposes the word can 
be understood as an approximate synonym for “immedi- 
ately causal:”” 

P4 In general, semantic relations * to the world 
(including reference) are not effective. 


'Ihe presence of a semantic relation cannot be causaliy de- 
tected at cither end—neither at the signifying object or 
event (term, word, representation, data structure, thought, 
etc.) nor at the signified entity (reference, denotation, or 
represented entity or state of affairs). You cannot tell what 
a representation represents, for example, by local mea- 
surement or detection. No physical detector, located at an 


12. Technicaliy, “effective” should be understood as a higher-order 
property, holding of those properties in virtue of the exemplification 
of which something can do physical (causal) work. That said, the rela- 
tion between computer sciences notion of effectiveness and classical 
treatments of causality warrants investigation—especially once it is 
recognized that the notion of effective computability cannot be de- 
fined purely abstractly, which 1 believe, See my “Solving the Halting 
Problem, and Other Skullduggetry in the Foundations of Comput- 
ing, unpublished manuscript. 

13. By semantic relations I mean the relations between signs (terms, 
noun phrases, representations, descriptions, thoughts, data struc- 
tures, etc.) and the objects that they signify (refer to, denote, describe, 
represent, or are about). 


object, can determine whether that object is the object of 
a description or directed thought. While “being referred 
to or represented” is a perfecty real property (which of- 
ten matters a great deal), no wave of discriminable energy 
travels along the arrow of intentional directedness; it is not 
a signal that any physical instrument can pick up. Crea- 
tures on Andromeda could be thinking about us right now, 
without there being any way for us to know or fınd out; 
similarly, you can refer to the sun without eight seconds 
having to pass before the sun is referred to. Nor can ref- 
erence (aboutness) be blocked by physical barriers. I can 
think about Alpha Centauri even if you lock me in a lead 
vault. 1£, in that vault, I suppose that Reno is east of Los 
Angeles, I can still be wrong. * 

'Ihe noneffectiveness of reference and semantics” is fun- 
damental to the nature of representation and intelligence, 


14. 1 describe the following iPhone application to my students, guar- 
antecing— with my life —that if they can build it, they will earn a 
billion dollars. The application is specified to work as follows: once 
someone loads it onto their phone and starts it up, the phone will 
beep every time someone thinks about them, 

Many students, being committed mechanists, claim that they 
could surely build such an app, at least conceptualiy, by instrumenting 
everyone brains, uploading everyone thoughts to the cloud, sending 
a signal to the targets phone at just the right time, and so forth. But 
that is not the point—nor the challenge. My claim is that there isa 
fact of the matter, right now, as to whether any particular person is be- 
ing thought about at any given instant —but that fact does not result 
in any physical disturbance or signal” arriving at that person. 

Tam not worried. 

15. İn saying that semantics is not effective, I am not denying that 
semantical acts can and do have causal conseguences. “Can you reach 
the salt?” may trigger the salt's being passed; Galileos defense of he- 
liocentrism led him to be branded as a heretic. Ihe point is that in- 
tentional acts neither make not reguire effective (causal) contact with 
the states of affairs they are about. 


and radically affects ontology as well—the objects and 
properties in terms of which we fınd the world intelligible. 
It is a conseguence of the most basic fact in all of phys- 
ics: that causal influences fall away locally in both space 
and time. The fact that the universe is a web of putely lo- 
cal influence poses a formidable challenge to any system 
or creature aiming to comprehend its world. It means that 
no system, human or machine, can determine what is go- 
ing on merely by “looking out” and seeing or sampling it 
(drastically limiting the applicability of Brooks's famous 
maxim that “the world is its own best model”“). Most of 
what matters, most of what an agent cares about—ev- 
erything in the past, everything in the future, everything 
that is far away, and a huge number of facts about what is 
dose by, too /—cannot be directly detected. Whether the 
news is true, whether the chair needs to be moved inside, 
whether you will ever come to appreciate the advice you 
were given—none of these things can simply be “read off” 
the impinging field of causal disturbance. 

Yet even if we cannot directiy detect it, we need to 
know—if we are intelligente—what is going on, distally: 
where we stored those acorns, that a predator lurks behind 
that rock, that the person who just left the room still exists, 
rather than having metaphysically vanished, that the sun 
will rise again tomorrow. If intelligence involves finding 


16. Rodney Brooks,“Intelligence Without Reason,' MIT Artificial In- 
telligence Laboratory Memo 1293 (1991); the phrase also occurs in the 
version of his “Intelligence Without Representation” in John Hauge- 
land, ed., Mind Design TI: Philosophy, Psychology, Artificial Intelligence 
(Cambridge, MA: MIT Press, A Bradford Book, 1997), p. 405. The 
point in the text is not that Brookss thesis is false; it is that it is only 
useful with respect to immediately surrounding circumstances, 

17. All those states of affairs that involve the exemplification of non- 
eflective properties, 


the world as a whole intelligible, which I assume to be the 
case, but we are in effective contact with only that minus- 
cule fraction that is nearby (the effective facts within a / 
r> spatiotemporal envelope), then it is fundamental to in- 
telligence to understand what is going on beyond what is 
effectively proximal. Put it more strongly: the world's being 
beyond effective reach is the only reason we need reasoning 
at all, As Strawson pointed out, we need to“know that our 
senses fail, rather than that the world fades.” 

How do we orient appropriately to a world beyond 
effective reach? By making use of locally effective struc- 
tures and processes in the world's stead—that is, by 
representing the world. This is essentially a metaphysical 
reguirement on intelligence.” 'Ihat it does not contravene 


18. P E Strawson, Imdividuals (London: Methuen, 1959). Similarly, 
“We think so that our hypotheses can die in our stead," Popper fa- 
mous maxim, makes sense only because of the discrepancy between 
the semantic reach of our thoughts and the causal extent of the world. 
19. Representation is widely decried, these days, throughout cognitive 
science and much of the humanities. Ihese antirepresentationalism 
sentiments are atrayed against a very narrow notion of representa- 
ton, however, thereby discarding one of the deepest insights into 
what reasoning and intellişence are. Acknowledging the tension 
between what is effectively proximal and what is normatively per- 
tinent but physically distal does not reguire treating representation 
as necessarily symbolic, pledging allegiance to naive realism, denying 
that ontology depends on culture or practice, asserting that cognitive 
activities are exclusively rational or deliberative, affırming a strict cor- 
respondence between the structure of the sign and the structure of 
the signified (a common interptetation of the Wittgensteinian “pic- 
ture theory” of representation), or a host of other presumptive ills. 
Representation as I am characterizing it here is as pertinent to enac- 
tive, feminist, poststructuralist, and ecoloğical views of cognition as to 
those traditionally labeled as intentional or representational. See my 
“Rehabilitating Representation” (unpublished manuscript presented 
at the Center for Cognitive Science, University at Buffalo, State Uni- 
versity of New York, Spring Colloguium, 2008). 


Pa (the noneffectiveness of semantics) stems from the fact 
that representations have two criterial properties: (i) they 
stand in noneffective semantic relations with what is dis- 
tal, with what is not immediately eftectively available (what 
they represent); and (ii) they can nevertheless effect or be 
affected by physical changes within causal proximity. Çua 
organized pattern of ink on a surface, a sign inscribed with 
“INTERSECTION AHEAD” can reflect light differentialİy, caus- 
ing signals to be activated in the retina, in tum triggering 
neural activity causing braking behavior; at the same time, 
within the regime in which it functions as a representation, 
the sign can stand in some kind of structural relationship 
(fashioned in such a way as to normatively guide action) 
with a state of affairs out of sight around the corner. So 
too a thought about an upcoming speed trap can helpfully 
cause you to slow down before the police can detect your 
approaching vehicle— that is, before you are within effec- 
tive reach of the radar. 

'Ihe use of representations with these coordinated 
dual properties is what allows an intelligent system to be 
physically possible (Pı). But in order for such symbols and 
thoughts about what is beyond effective reach to be correct 
or wortbwhile, those“within reach” local, effective structures 
and the processes defined over them must be governed by 
norms framed in terms of the distal,“out of reach” (nonef- 
fectively available) situations that the system is about. © To 
putitin a single sentence: 


20. Soundness and completeness are the specific versions of these 
norms appropriate to the particular assumptions and restrictions ap- 
plicable to formal logic. 


9 REPRESENTATIONAL MANDATE: The proper func- 
tioning of any world-directed system— any system that 
is thinking about or representing or processing informa- 
tion about tbe world—must be göverned”” by norma- 
tive criteria (P3) applying to its mechanical operations 
(Pı) that are framed in terms of situations and states 
of affairs in the world that the system is representing 
or reasoning about (P2), which sityations and states of 
affairs will not, in the general case, be within effective 
(causal) reach (P4). 


Because the norms are framed in terms of the world, and 
govern the local mechanism— that is, because what mat- 
ters, in order for the reasoning to be proper or worthy, is 
what is going on in the world—I say that reasoning sys- 
tems, and all of intelligence, must be deferential. 

'Ihe impact on ontology is direct. Representations can 
satisfy distal norms (norms defined in terms of distal and 
noneflectively accessible states of affairs) only if their lo- 
cal, effective properties correspond appropriately”” to the 
properties of the represented situations in whose stead 
they serve, It follows that the world, to be intelligible, must 
manifest some degree of time- and distance-spanning cor- 
relation.* At any given place and moment in time, smoke 
can be recognized as a signal of fire only if the causally efti- 
cacious (i.e., effective) presence of smoke reliably correlates 


21. See note 11 (p. 11). 

22. Appropriate correspondence” may include structures that medi- 
ate activity—or even such activity itsel—so long as it leads to be- 
havior normatively appropriate to that which is thereby represented. 
23.1 take the guantitative, nonsemantic conceptions of information 
espoused in computer science and physics to be (postontological) 
measures of the world's correlation. I do not believe such correlations 
should be called information, but they are undeniably important. 


with the (potentially unobservable, and thus at least poten- 
tally not causally efficacious) fact of something burning, 
perhaps distally. If reality was everywhere independent, 
the universe would be random, tepresentation impossible, 
and intelligence nonexistent. At the same time, the world is 
not completely correlated; that would lock it up into a rig- 
id dutch, with representation, intelligence, and creatures 
again proscribed.”* At any level at which we find it intel- 
ligible, the world must be partially connected and partially 
disconnected.” 


Every great idea languishes for most 
of history in obscurity, has one brief 
moment of glory, and lives out its 


dying days as a platitude.”* 


Figure 1 


24. See the discussion of the “gear world” in On the Origin of Objects, 
199—200. 

25, Some may be tempted to phrase this point in terms of degrees 
of freedom, but any such measure reguires ontology, and ontology 
rests on this pattern of partial connection and disconnection, not the 
other way around. It might merit investigation, though, to explore 
what this “balance” between freedom and correlation that is so crucial 
to registration, information, fınding the world intelligible, etc., is due 
to physical laws, constitutive conditions governing the abstractions 
we employ in our registrations, or other factor/s). 

26.1 heard this saying from my father when I was a child, Whether it 
was original to him or whether he was guoting someone else | do not 
know, but I have been unable to determine another source, 


'Ihe patterns of connection and disconnection neces- 
sary for representation, and hence for intelligence, in turn 
put conditions on possible ontologies, It is no accident that 
your personhood is not contingent on the moment-to-mo- 
ment orientation of your limbs—that you do not go out 
of existence when you stand up, with a new person magi- 
cally springing into being. If that were true—if our concept 
of “person” constrained limb orientation—I would not be 
able to know, at any given moment, whether you existed. 
In fact no one”? would have a continuing identity at all. It 
is not just that intelligence reguires the ability to track and 
think about objects, in other words; objects must be patches 
of reality that can be thought about and distally tracked. Epis- 
temology constrains ontology; it is not just the other way 
around.“ 


Ihat thinking, intelligence, and information processing 
satisfy criteria Pı—P4 is the fundamental idea behind for- 
mal logic and computing. Ihat they are not just things 
that humans do, but also things we can build automatic 


27. Ihat is, no space-time worm that we, in our current ontology (reg- 
istration scheme; pp. 35—36), register as a person. 
28. Realists need not panic. Ihe point is not that epistemological or 
psychological states somehow magically reach out and mysteriously 
adjust the metaphysical plenum. Nor is it merely that registrational 
practices are genuine activities in the world (though they are). Rather, 
as I put it below, our ontological (or as 1 put it, registration”) schemes 
must be such as to support epistemic access to their objects—must 
parse reality in such a way that the resulting “objects” can be referred 
to and thought about. What it is to be an object, in other words, is 
to be a possible locus of registrational accountability. Yes, an object is 
different from our representation of it; but that does not imply (or 
reguire) that ontology and epistemology are logically independent. 

I am using 'constrain,” here, along the lines of the “govern” in the 
REPRESENTATIONAL MANDATE (p.17). 


machines to do, was a stunning discovery in the first sev- 
eral decades of the twentieth century. By now it is so obvi- 
ous that it barely garners mention (see figure 1, page 18). 
But the insight is fandamental—and, being a precondition 
for the very possibility of intelligence, impossible to imag- 
ine not being true. It enabled the study of formal logic, un- 
derwrote the development of the computer, unleashed the 
first generation of Al systems, led to the development of 
knowledge representation languages, and in general shaped 
our conception of how to build world-directed computer 
systems. 

Ina moment I will argue that this general insight un- 
derlies second-wave Al, too. The initial idea about how 
to manifest it in a machine, however—that is, the concep- 
tion underpinning GOFAI or first-wave Al (1 will use the 
two terms essentially interchangeably) — was more limited 
than initialiy realized. In the GOFAI era, it was assumed 
that the evident way to implement it was to build an inter- 
connected network of discrete symbols or data structures, 
organized in roughly propositional form, operating accord- 
ing to a regimen dictated by an egually symbolic program. 
Ihis dassical “symbolic” architecture still underlies such 
staple programs as health record systems, Facebook data- 
bases, and the calculations of the Internal Revenue Service. 
Ihere is a sense in which it is still in use in all programs 


(such as Microsoft Word). 


* * * 


To a contemporary imagination, especially one entranced 
by the achievements of second-wave Al, the intellectual 
underpinnings of GOFAI and first-wave Al may not 
seem all that impressive— certainly not as groundbreaking 
as they did in the 1960s and 1970s. But it is essential to 


recognize that not only did they launch the whole Al en- 
terprise, and in fact the entire development of computing, 
but the insights on which they are based remain durabiy 
deep and powerful. If we are to draw a comprehensive map 
of the territory of Al, they need to be given appropriate 
and respectful place. 


3 — Failure 


Mevertheless, GOFAI failed. Or anyway it is deemed to 
have failed. The failures can be summarized as of four 
main types.” 


Fı Neuwrological: Ihe brain does not work the way 
that GOFAI does, All of today's mainstream CPUS$, 
and ali cdassical Al systems, consist of relatively few 
serial loci of activity (one to a few dozen threads) 
running at extreme speeds (on the order of 10” 
operations per second), pursuing relatively deep 
levels of inference or procedural nesting. So far as 
we know, the functioning of the brain is almost the 
complete opposite. Brain operations are roughly 
so million times slower, deriving their power from 


massive parallelism.” 


1, GOFAT was not abandoned. Research in its vein has continued; 
in chapter 7 I argue that it addresses issues crucial to forward prog- 
ress, But GOFAIJ no longer has imaginative grip as “how intelligence 
works”; it is widely disparaged in cognitive science and neuroscience; 
the center of gravity of Al thinking has shifted to machine learning 
and other second-wave approaches; and, as 1 argue here, a number of 
its founding assumptions (Cı-C4) are untenable as a general model 
of intelligence. 

2. This typology is loosely based on Part LI of Dreyfuss landmark 
What Computers Can't Do, 1972. 

3. According to what is known as Feldmans “1oo-step rule” (Jerome 
Feldman and Dana Ballard, “Connectionist Models and their Prop- 
erties,” Cognitive Science 6, no. 3, 1982, 206), the brain can compute at 


F2 


Whether these low-level architectural differ- 
ences are germane to the achievement of human- 
level intelligence, however, is not a straightforward 
guestion. To answer would reguire knowing what it 
takes to have intelligence like ours, what intelligence 
of anonhuman kind might be like, what sorts of im- 
plementation strategies the brain uses, and so forth. 
Nevertheless, GOFATS decline has led to renewed 
interest in “brain-style” computing, of which today's 
“neural networks,' deep learning systems, and other 
machine learning architectures are instances.” 


Perceptual: Most GOFAI theorists thought that 
perception—recognizing or “parsing” the world 
based on perceptual input from sensors— would 
be conceptually simpler than simulating or creat- 
ing'real intelligence." After all, in Descartes's phrase, 
“mere beasts” are good at it, in some cases notably 
better than people. Plus, the ontological assumption 
underlying GOFAI (C4; also Fa, below), accord- 
ing to which the world consists of unambiguous 
objects exemplifying discrete properties, suggested 
that perception would merely involve determining 
what objects were “out there," what properties they 
manifested, what types or categories they belong to, 
and so on. 


It did not work out that way. The difficulty of 


most 100 serial steps per second—sobering, given the brains ability 
to perform many cognitive tasks in under a second. . 

4. Neurally inspired architectures have a long history, going back to 
McCulloch and o. And not all massively parallel architectures were 
neurally inspired—for example, some early semantic nets. But it is 
unarguable that the imaginative center of attention in Al has shifted 
from serial architectures to massively parallel networks, 


3 — Failure 


Otto Lowe, 2018 


Figure 2 


real -world perception was one of several profound 
humilities to which we have been led by our experi- 
ence with fırst-wave Al. 

When people first attached digital cameras to 
computers, they were Habbergasted. Ihis was the 
early 19705, after all. The idea of using electronics 
instead of fılm had just been bruited (in 1961 in re- 
gard to satellites; the first digital camera was not 
released to the market until 1975, toward the end 
of GOFATS reign). Why were people taken aback? 
Because what came through the sensors was a mess. 
It turns out that the idea that the world consists of 
a simple arrangement of straightforward objects 
is the delivery, to our consciousness, of the results 
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The Promise of Artificial Intelligence 


of an exguisitely sensitive, finely tuned perceptual 
apparatus running on a 100-billion neuron device 
with 100 trillion interconnections honed over 500 
million years of evolution. Figure 2 may seem like a 
straightforward picture of an artist” in a workshop; 
figure 3 is a rendering this artist once made to con- 
vey to human (visual) observers what he believes 
the world is actually like— that is, it is an image 
that, after human perceptual processing, gives us a 
rough sense of what the world looks like prior to 
human processing. 


ği Mi 


Adam Lowe, CR29, “Studio with phone,' 1993“ 


Figure 3 


5. Adam Lowe, of Factum Arte (http://www.factum-arte.com). 
6. Collection of Adrian Cussins.I used this image as chapter 10 of On 


tbe Origin of Objects. 
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Descartes was no fool. His standards on ratio- 
nality were much higher than commonly appreci- 
ated. But history suggests he was insuffıciently im- 
pressed with the achievement of looking out and 
secing a tree. 


Epistemological: Ihinking and intelligence, on 
the GOFAI model, consisted of rational, articulat- 
ed steps, on its founding model of logical inference, 
As Dreyfus insisted, and many cognitive theorists 
have emphasized,” in many situations a more accu- 
rate characterization of intelligence is one of skili- 
ful coping or navigation— of being “thrown” into 
individual and social projects in which we are em- 
bedded and enmeshed. Even thinking, these writ- 
ers argue, rather than consisting in a series of read- 
ily articulable steps, emerges from an unconscious 
background—a horizon of ineffable knowledge 


and sense-making. 


7. For example, Joseph Weizenbaum, Computer Power and Human 
Reason: From Judgment to Calculation (New York: W. H. Freeman 
and Company, 1976); Lucy Suchman, Human-Machine Reconfıgu- 
rations: Plans and Sityated Actions (Cambridge: Cambridge Uni- 
versity Press, 2007); Humberto Maturana and Francisco Varela, 
Autopoiesis and Cognition: The Realization of the Living (Dordrecht: 
Reidel, 1980); Terry Winograd and Fernando Flores, Understanding 
Computers and Cognition: A New Foundation for Design (Norwood, 
MA: Ablex Publishing, 1986); Eleanor Rosch, Francisco Varela, and 
Evan Thompson, The Embodied Mind (Cambridge, MA: MIT Press, 
1991); Evan Thompson and Francisco Varela, “Radical Embodiment: 
Neural Dynamics and Consciousness,' Trends in Cognitive Sciences 5, 
no. 10 (2001): 418—425; and Daniel Hutto," Knowing What? Radical 
Versus Conservative Enactivism, Phenomenology and the Cognitive 
Sciences 4, no. 4 (2005): 389—405. 


Fa Ontological: Ihe misunderstanding about per- 
ception, and perhaps about thinking as well, betray 
a much deeper failure of first-wave Al: its assump- 
tion, already mentioned (Ca), that the world comes 
chopped up into neat, ontologically discrete objects. 
In fact one of the primary theses of this book is that 
the misconception of the world in first-wave Al is 
the “smoking gun” that explains its ultimate inad- 
eguacy. İn chapter 6, I will argue that the successes 
of second-wave Al can be interpreted as evidence 
for the inadeguacy of formal ontology both as a ba- 
sis for intellişence and as a model of what the world 


is actualiy like.* 


Of these four types, the ontological issues cut the deepest. 
Ihey not only underlie the perceptual and epistemological 
difficulties, but also need to be appreciated in order to un- 
derstand how and why second-wave Al has made at least 
some progress toward interpreting the world, especially at 
the perceptual level. 


Ihree points are especially relevant. 

First is a general thesis with which few GOFAI design- 
ers would disagree, even if no GOFA!J system could itself 
be said to understand it. Though framed epistemicalIy (as 
having to do with intentionality), the point rests on un- 
derlying ontological facts. It is universally accepted that 
the world can be described or conceptualized in multiple 
ways—as is often said, at various “levels of description.” 
Reality itself is assumed, at least for all practical purposes, 


8. In chapter 7 1 argue that preserving what was good about GOFAT 
reguires extracting its insights from the presumption that they can 
be understood in terms of, and uniguely apply to, a world adeguately 
characterized in terms of formal ontology. 


to be surpassingiy rich, about which any ontological 
“parse” provides only partial information. Representations, 
descriptions, models, and the like all interpret or picture 
or filter the world through abstractions or idealizations— 
conceptual “frames” that highlight or privilege some aspects 
of what is represented, minimize (or even distort) others, 
and ignore or abstract away from a potentially unbounded 
amount of in-the-world detail? 

'Ihis commonplace is easiest to see in the case of con- 
ceptually structured representations—including not only 
propositional structures of the sort exemplified in logic 
(predications, conjunctions, disjunctions, negations, im- 
plications, guantifications, etc.), but also other represen- 
tational forms that characterize the world in terms of a 
pre-established stock of objects, types, properties, and so 
on, such as those used in computer-aided design (CAD) 
systems, architectural blueprints, databases, and the like.” 


9. Some philosophical readers may object that names, at least, refer 
to objects “without loss" —that is, to objects as such, and therefore do 
not impose even a trace of a conceptual frame. More later; for now 
I would just note that this would be true only if objects were ob- 
jects (i.e., had determinate identity conditions) independent of being 
taken as objects, a standard realist position with which I disagree. See 
On the Origin of Objects. 

10, Evans ties this notion of conceptuality to what he calis a“General- 
ity Condition” (Gareth Evans, Varieties of Reference, Oxford: Oxford 
University Press, 1982, 100—105):a reguirement that for a system to 
be able to entertain a “thought” or representation that 4 is f, it must 
also be able to entertain the thought that b is f, cis f, etc., for any terms 
b, c, etc. that represent other objects, and also that g isg, gis h, etc, 
for any predicates g, h, etc. representing other properties (modulo ap- 
propriate type testrictions). This is the sense of conceptuality that 
underwrites the characterization of the nonconceptual given in the 
sidebar on the next page; it differs from the one McDowell critigues 
in Lecture III of Mind and World (Cambridge, MA: Harvard Uni- 
versity Press, 1996). 


Nonconceptual Content 

“Nonconceptual content” is a philosophical term used to explain the 
fact, evident upon reflection, that we are capable of thoughts and 
judgments that, although grounded in the ways of the world, and 
capable of being right or wrong, are not framed (in our minds) in 
terms of anything like a discrete set of articulable concepts. Classic 
examples include the speed at which one rides a bicycle, the position 
in ones egocentric personal space where items are located, shades 
of color, and so on. Adrian Cussins, an early theorist of the notion, 
when stopped for speeding on his motorcycle, famously answered a 
policeman guestion about whether he knew how fast he was going 
with the statement “In some ways yes; in some ways no” (personal 
communication). Evans (MIT seminar 1978) reported on a footstep 
behind him when working in his study; in one sense he knew with 
exguisite precision where it was, yet would have been entirely unable 
to describe that location precisely in publiciy shared concepts— feet, 
angle in degrees, and the like (see Gareth Evans, Varieties of Refer- 
ence, 105). Similarly, it would be of limited help, if you were having 
trouble with your second tennis serve, to suggest that you hit the ball 
at the speed at which copies come out of your office copier. 


But it also applies to most analog representations (those 
that represent continuous guantities in the represented 
domain—the domain of the problem to be solved, typi- 
cally — with analogous teal-valued guantities in the rep- 
resentational medium), since the property-to-property 
correspondence in such cases is typically assumed to be 
discrete at a higher-level of abstraction." 


11. As analog computers demonstrate, characterizing a situation in 
terms of discrete properties and relations does not mean that the val- 
yes of those properties need to be discrete. The use of the differential 
calculus in classical physics is fully conceptual, on this account, even if 
any given velocity or acceleration or mass can be measured with a real 


3 — Failure 


Figure Pi 


'Ihe second point is also traditionally framed epistemo- 
logically. As noted in Fa, theorists in numerous fields have 
emphasized the fact that not all human understanding has 
a "conceptual” form at all. The point is most familiar from 
phenomenological theorizing, but similar views under- 
write several contemporary currents in cognitive science, 
induding enactivism, connectionism, and deep learning. 
Analogous intuitions have been explored in analytic phi- 
losophy in the notion of nonconceptual content (sidebar on 
the previous page). These approaches are united in viewing 


number. What matters is that the properties of velocity, mass, etc, 
are discrete. There is no such thing as something “halfway between a 
mass and charge," though even that might be expressible in terms of 
a higher-order discrete categorization (John Haugeland, "Analog and 
Analog,” Philosophical Topics 12, no. 1, 1981, 213—225). 

12. A masked version of figure 6, p. 34. 
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The Promise of Artificial Intelligence 


Figure 5 


intelligence as emerging against an unconscious back- 
ground—as 1 put it above, against an ineffable horizon of 
tacit knowledge and sense-making. 

Together with the perceptual points made earlier, these 
views provide indirect support for an ontological world 
view I have sketched elsewhere,? for which figures 4—6 
provide a metaphor. Figure 4 (previous page) is a pho- 
tograph of some islands in Ontarios Georgian Bay. Al 
ready, one can see that the real-world topography fails to 
support the cut-and-dried ontological registration that 
GOFAIJ assumed. Figure 5'* “cleans the picture up” in a 


13. On the Origin of Objects. 

14. Figure 5 is by design simplistic; the filters used to create it are 
not especially fine-grained. One could construct an image in which 
they encoded vastly more detail —as, for example, is current prac- 
tice in digital GIS systems. But the conceptual point would remain: 


way reminiscent of GOFAI knowledge representation— 
making the islands, though still relatively detailed, “clear 
and distinct,” and also internally homogenous, in the way 
in which conceptual models (such as data bases) all ulti- 
mately assume. While the guestion of “how many islands 
are there” may have a determinate answer in figure 5, the 
same is not true of the world depicted in figure 4. Distinct- 
ness flees, as realism increases. In the world itself, the gues- 
tion lacks a determinate answer, 

More telling yet, though, is figure 6 (next page)—the 
same photographic image as figure 4, except now reveal- 
ing the submarine topography. Compared to the world's 
messiness, the image is still simple: gravity is a single di- 
mension of salience, the water line is relatively sharp, the 
image is gray scale, and so on. Nevertheless, if the islands in 
the image are taken as analogs for properties, then the im- 
ages suggest what in fact is true: that as soon as one presses 
for detail, distinctions multiply without limit. 


* * * 


any (fırst-wave) knowledge representation structure is formulated in 
terms of a well-specifled particular level of granularity; the represen- 
tation “individuates” the world at a specific level of grain—no fıner, 
In figure 5, as in all conceptual representations, there is a definite 
fact of the matter about how many islands there are, where exactly 
the boundaries are specified as being, etc. One would not learn more 
about the islands by studying the representation in any finer detail 
than necessary to reveal those facts. (This is what Dretske nonstan- 
dardiy means by “digital” representation, as opposed to “analog”; see 
his Knowledge and the Flow of Information. Cambridge, MA: MIT 
Press 1981, 135—141.) 

Someone might argue that the same is true of (even analog) pho- 
tographs, because of the ultimate granularity of fılm. What is crucial, 
though, is the same is not true of the reality that these images represent. 
'Ihe world itself is arbitrarily detailed. 
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Paul Bennett Photography 
Figure 6 


Ihough our concepts and the properties they represent 
may seem discrete, in other words, the failures of GOFAT 
and the merit of the critigues mounted against it suggest 
that the idea of a "clear and distinct” world is an artifact 
of how it is represented. The issue for Al is that, in or- 
der to function in the world, Al systems need to be able 
to deal with reality as it actually is, not with the way that 
we think it is—not with the way that our thoughts or lan- 
guage represent it as being. And our growing experience 
with constructing synthetic systems and deploying them 
in the world gives us every reason to suppose that “beneath 
the level of the concepts'—beneath the level of the objects 
and properties that the conceptual representations repre- 
sent—the world itself is permeated by arbitrarily much 
more thickly integrative connective detail, Tt is not just that 


our concepts sometimes have vague or unclear boundar- 
ies; it is that these facts tell on a world that itself is not 
itself clear cut. (Is a boisterous child the same as or differ- 
ent from a rambunctious child—or a boisterous CEO? If 
we are climbing the tallest peaks in Canada, and there is 
another local maximum 100 meters away from the summit 
we have just reached, do we need to go over there as well? 
Where does one “fog” end and another start? Reality will 
not tell us. If we want'clear and distinct” answers, we need 
to employ conceptual schemes that impose them.) 


What are the conseguences of these insights for AI? What 
follows from recognizing that the nature of reality is as 
suggested in figure 6: a plenum of surpassingily rich differ- 
entiation, which intelligent creatures ontologically “parse” 
in ways that suit their projects? 

Some of the technical implications will be explored in 
chapter 5. Overall, though, it means that Al needs to take 
on board one of the deepest intellectual realizations of the 
last 50 years, joining fields as diverse as social construction, 
guantum mechanics, and psychological and anthropo- 
logical studies of cultural diversity: that taking the world 
to consist of discrete intelligible mesoscale objects is an 
achievement of intelligence, not a premise on top of which 
intelligence runs. Al needs to explain objects, properties, 
and relations, and the ability of creatures to fınd the world 
intelligible in terms of them; it cannot assume them.” 

How we register the world, as I put it“ find it onto- 
logically intelligible in such a way as to support our projects 


15. Ontology reguires naturalization too, to put this in philosophical 


terms. 


16. See On the Origin of Objects, and“Rehabilitating Representation” 
(discussed in chapter 2, page 15. note 18). 


and practices—is in my judgment the most important task 
to which intelligence is devoted. As 1 will describe in more 
detail in subseguent chapters, developing appropriate reg- 
istrations does not involve merely “taking in what arrives at 
our senses, but—no mean feat—developing a whole and 
integrated picture accountable to being in the world. It is 
not just a guestion of fınding a registration scheme” that 
“fıts” the world in ways locally appropriate to the project 
at hand, but of relentless attunement to the fact that reg- 
istration schemes necessarily impose non-innocent ideal- 
izations—inscribe boundaries, establish identities, privi- 
lege some regularities * over others, ignore details, and in 
general impose idealizations and do an inevitable amount 
of violence to the sustaining underlying richness. This 
process of stewardship and accountability for registration, 
never imagined in the GOFAI project, is of the essence of 


intelligence.” 


* * * 


One can (people do) say that GOFAJ misconceived cogni- 
tion, but as the foregoing discussion makes clear, I believe 
the deeper problem is that it misconceived the world. Why 
did it do so? One thing can be said, involving a subtlety 
that bedevils all discussion of Al systems'engagement with 
the world. 

'Iwo “views” or registrations of the world are relevant to 


17. Ihis process is sometimes described as identifying an appropri- 
ate “level of description” or “level of abstraction” (sometimes even as 
“coarse-graining”), but the metaphor of"levels” can be misleading. 

18. 1hat is: the metaphysical warrant for that which we register as a 
regularity. 

19. Ihe failure to imagine and therefore to take up these issues is a 
major reason, | believe, for GOFATS inability to make progress on 
common sense, See the sidebar on the next page. 


Common Sense 

GOFATS ontological presumptiveness, its blindness to the subtleties 
of registration, and its inadeguate appreciation of the worlds rich- 
ness are the primary reason, in my judgment, for its dismal record 
on common sense, Notorious examples of early failure in this regard 
indude systems suggesting boiling a kidney to cure an infection, and 
attributing reddish spots on a Chevrolet to a case of measles.” 

The initial response to shortfalls of this sort, most famously 
taken up in the CYC project,” was two-pronged: fırst to codify, in 
a İogical representation, all common knowledge to be found in an 
encyclopedia, and then, when that proved insuffıcient, to encode all 
common knowledge not in an encyclopedia.Ihose projects, and the 
wider “commonsense reasoning” program of fırst-wave Al, though 
still pursued in some guarters, have largely faded out of sight as 
models on which to base commonsense reasoners. Ihey live on in 
such structures as Googles “Knowledge Graph,' which is used to 
organize and provide short answers to gueries in search engines. 
Tellingiy, though, the results of such searches are snippets for inter- 
pretation by human intelligence, not the basis of the machines own 
intelligence." 


a, Hamid Ekbia, Artificial Dreams (Cambridge: Cambridge University Press, 
2008), 96-97. 

b. See http://www.cyc.com, where this project lives on. 

c. See my “Ihe Owl and the Electric Encyelopaedia,” Artificial Intelligence 47 
(1991): 251—288. 


the design or analysis of an Al system, or indeed of any 
intentional system at all, incuding people. The first is the 
designer's or theorist's registration of the world in which 
they are building the system, and into which they will de- 
ploy it, or in which they are analyzing a system or crea- 
ture—that is, the world, as registered by the theorist or 


designer, that they expect that system to fınd intellişible, 
deal with, behave appropriately in. Ihe second is the sys- 
tems own registration of that world—the registration that 
will presumabiy lead it to behave and deal with the world 
in the way that it does. There is no reason to suppose the 
two registrations will necessarily align. As explored in greater 
detail in the sidebar on the following pages, the sugges- 
tion about why GOFAJ may have misconceived the world 
is this: because first-wave Al and GOFAI took a techno- 
scientifıc attitude to their subject matter of constructing 
an intellişent system—analyzing it in terms of its causal 
or mechanical constituents formalIy conceived”* —and be- 
cause they did not deal with the phenomenon of registra- 
tion at all, they may simply have assumed that intelligence 
could be constructed in a system that itself took a techno- 
scientific attitude to its world. 
It did not work out that way, 


20. For example, analyzing it in terms of what Sellars would call the 
“scientific” image, as opposed to the manifest image likely used by the 
system being modeled. 


3 — Failure 


Theorist vs. Subject Registration 

An issue that comes up when theorizing intentional systems is the 
relation between (i) the theorists registration of the world toward 
which the system in guestion is oriented, and (ii) the registration of 
the world in terms of which that system fınds that world intelligible. 
Take T,S,and W to be the theorist, system under investigation, and 
world or task domain, respectively. If 'T is a simple realist, taking W 
simply to be “is as it is, then the issue for T is straightforward: T can 
characterize W, or at least those aspects that S deals with, correctiy," 
and assume that S does— or should do so—as well. 


If registration is taken seriously, however—as we must in order 
to do justice not only to AJ, but to the world and intentionality more 
generally—then at best it would be preemptively inscriptive, and 
at worst false, for T to presume that S's registration matches their 
own. T's registration may be adeguate with respect to nonsemantic 
aspects of Ss engagement— that is, adeguate in terms of which to 
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understand S's causal interactions with W (except in as much as 
those causal interactions need to mesh with S's intentional registra- 
tion, such as serving as satisfaction conditions for S's planned ac- 
tions). A much-touted feature of dynamical systems accounts of 
cognitive behavior, for example, is their ability to theorize the agent— 
world relation in the form of the differential eguations reguired by 
dynamical systems theory. 

When it comes to intentional or semantic relations between S 
and W, however, the situation grows more complex—as depicted in 
the figure on the previous page. Three (related) issues are at stake: (i) 
how T registers W, (ii) how S registers W, and (iii) how T registers 
S5 registration of W. 

Although this is not the place to engage in a detailed technical 
analysis of how these registrations can be meshed or appropriately 
coordinated, a few remarks can be made. First, to the extent that 
T's and Ss registrations align, the issue is unlikely to be problem- 
atic. Ihe problem grows more challenging to the extent that they 
differ—akin to that of understanding a foreign culture or alien spe- 
cies. There is no reason to suppose that T will necessarily be able to 
“grasp, at all, how it is that S registers W, no matter how committed 
'I may be to doing so. 

One might imagine that understanding human intelligence would 
be an instance of the former case, and therefore straightforward, but 
that is where the issue mentioned in the text comes to the fore, I£ T 
brings a scientific or technical theoretic attitude to bear on its theory 
of S, S$ intentionality, how S perceives and thinks about the world, 
and so on, then T's registration of W is perforce likely to be scien- 
tific—treating W in terms of formal ontology: discrete objects, ex- 
emplifying cearly-defined properties and standing in unambiguous 
relations, and so on. Unless T is theorizing how S is a scientist, how- 
ever, 1's scientific registration of W and Ss nonscientific registration 
are liable to part company. 


These discussions illustrate one way to understand how fırst- 
wave Al and GOFAJ may have 'got the ontology wrong." It is easy 
to see how, given that GOFAJ was blind to registration overalı, it 
would have made an implicit presumption that elementary intelli- 
gence could be fashioned out of the GOFAIJ researcher's scientific or 
technical attitude toward W. Or put another way, GOFAJ may have 
failed (ontologicalIy) because it projected its own default #heoretic at- 
titude onto S. 

I take it to be a conseguence of the metaphysical position de- 
scribed in On the Origin of Objects that no deferential or accountable 
registration can be constructed from such a base. 


4 — Transition 


So GOFAIJ failed. Beginning around 1980, a variety of 
new approaches took hold. One, to be considered in the 
next chapter, shifted attention “downward.,' to consider the 
brain and brain-style computational architectures. Ihis is 
the architectural approach that underlies Al's headlong 
embrace of machine learning and second-wave Al. It has 
developed in parallel with explosive growth in cognitive 
neuroscience, now arguabiy the most important sector of 
cognitive science. Together, these two projects currently 
have something of a grip on the intellectual imaginations 
of both Al and cognitive science. 

Before second-wave Al took hold, however, four other 
“broadening” approaches gained prominence: 


Embodied . Take the body seriously. 

Embedded. Take context and surrounding situation 
seriously. 

Extended . Maybe the mind is not just in the brain, 
or even brain plus body, but extends into 
the environment (which we, both individu- 
ally and as societies, arrange and construct 
so that it can serve as a form of "cognitive 
scaffolding”). 

Enactive : Dont separate thinking from full-blooded 
participation and action. 


Ihe embodiment thesis not only involves consideration 


of the whole body (limbs, torso, etc.) of the organism in 
guestion, and takes activity to be an integral component of 
intelligence and cognitive skill (for example in perception 
and navigation), but it also focuses on the physical proper- 
ties and restrictions on the brain or processing unit (such 
as heat, energy usage, etc.). 

Ihese approaches were most vigorously defended dur- 
ing the 1980s and 1990, in a period that has come to be 
known as the“Al winter”"—a term descriptive of a lull in 
the Al funding environment, not per se a comment on the 
pace of intellectual development, though not unrelated to 
the failure of GOFAI to reach its promises, The four the- 
ses were primarily advocated in cognitive science, but they 
also garnered some prominence in Al, where they con- 
tinue to garner allegiance. Ihey have not been embraced 
as such in the software engineering world, but if explicitly 
articulated might be endorsed by many members of that 
community—so long as the “body” of the computer were 
understood to be something like the CPU and directiy af- 
fıliated memory, or at least the “local machine. Certainly 
their overall sense is manifested in the rapidiy expanding 
networks of the world's growing computational infrastruc- 
ture (e.g,, in the widely trumpeted “internet of things”). 

I agree that human intelligence is embodied, embedded, 
(somewhat) extended, and (often) enactive—and that all 
four are critical considerations in cognitive science. But I 
think the reasons for GOFAT' failure go deeper than its 
failure to deal with these issues, 

In brief, the reasons are three. The first has already been 
suggested: GOFAT's untenabiy rigid view of formal ontolo- 
gy. Ihe second has to do with the semantic point made ear- 
lier (P4): that reference, representation, and semantic rela- 
tions to the world are not effective, and therefore cannot be 


“seen” from a standpoint of blanket mechanism. The third 
arises from a failure to recognize the profound importance 
of taking our thoughts and representations and informa- 
tion to be aboyt the world. We need to understand the con- 
ditions on world— what the world is, and what it is to hold 
something accountable to being in it, to being hosted by it. 


s5 — Machine Learning 


Fast forward, then, to the present—to deep learning and 
afhliated machine learning (ML) technologies associated 
with second-wave Al. These systems have made definite 
progress on the first and second of GOFATS failures (neu- 
rological and perceptual) —and have arguabiy begun to ad- 
dress, though they have by no means yet fully embraced, 
the third or fourth (ontological and epistemological). 
ML is essentially a suite of statistical technigues for: 


ı. the statistical cassification and prediction of patterns 
2. based on sample data (often guite a lot of it) 
. using an interconnected fabric of processors 


v3 


4. arranged in multiple layers. 


Ihese technigues are implemented in architectures often 
known as “neural networks,' because of their topological 
similarity to the way the brain is organized at the neural 
level, Figure 7 illustrates a way in which they are often de- 
picted, but a better way to understand contemporary ma- 
chine learning is in terms of the following four facts. 


ı. Ihough the phrase'machine learning was employed in era of fırst- 
wave AL I will use it here, especially the'ML'acronym, in its contem- 
porary sense: to refer not only to deep learning algorithms but also 
to a variety of follow-on technologies, induding deep reinforcement 
learning, convolutional neural networks, and other technigues involv- 
ing statistical computations over complex graph confıgurations. 

For deep learning in particular, see Yann LeCun, Yoshua Bengio, 
and Geoffrey Hinton, “Deep Learning” Nature 521, no. 7553 (2015): 
436—444. 
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Figure 7 


Dı Correlations: As we have seen, fırst-wave (GOFAT) 
systems were built to entertain and explore the 
conseguences of symbolically articulated discrete 
propositions implemented as formal symbols rep- 
resenting objects, properties, and relations in terms 
of a presumptively given formal ontology. Based on 
this model, rationality and intelligence” were taken 
to involve deep, many-step inference, conducted by 
a serial process, consisting of one or a few threads, 
using modest amounts of information, formulated 
in terms of a small number of strongly correlated 
variables (sidebar, next page). Standard logical 
connectives, such as negation (-—), conjunction 
(4), disjunction (v), implication (5), and the like, 
procedure and class definitions, and so on, can be 
understood as various forms of 100 positive and 


2. Or thought, or cognition—as indicated earlier, no distinctions 
were being drawn at the time. 


negative correlation. Ihe model makes sense under 
the dassical assumption of formal ontology, partic- 
ularly under the grip of Descartes's desideratum of 
“clear and distinct ideas.” 

As indicated in the sidebar, below, contemporary 
machine learning is essentially the opposite. It con- 
sists of shallow (few-step) inference conducted by a 
massively parallel process using massive amounts of 
information, involvinga huge number of weakly cor- 
related variables. Moreover, rather than 'exploring 
the conseguences” of such correlations, its strength 
is to learn and reproduce mappings between inputs 
and outputs. Whether the mappings should be un- 


derstood as relating causal patterns in the machine 


GOFAI vs. Machine Learning 

Ihe most compact way to understand the difference between 
GOFAJ and machine learning is in terms of their opposing positions 
on five conceptual axes. 


GOFAI| 


1. Deep (many-step) inference 

2. Byaserial process, using 

3. Modest amounts of information 

4. İnvolvinga relatively small number of 
5. Strongly correlated variables 


Machine Learning 


ı. Shallow (few-step) inference 

2. Bya massively parallel process, using 
3. Massive amounts of information 

4. İnvolvinga very large number of 

. Weakly correlated variables 


Ya 


(i.e, as uninterpreted mechanical patternings) or 
complex representations of confıgurations of the 
world (i.e, as interpreted) is a guestion we need 
to examine, Most literature appears to discuss it 
in terms of mechanical confıgurations, though the 
critical probabilities are always understood in terms 
of what is represented. 

What is called “face recognition” is widely touted 
as an ML success. But like many other terms un- 
critically applied to computational systems, the 
term “recognition” rather oversells what is going on. 
A better characterization is to say that ML systems 
learn mappings between (i) images of faces and (ii) 
names or other information associated with the 
people that the faces are faces of. We humans often 
know the referents of the names, recognize that the 
picture is a picture of the person they name, and so 
forth, and so the systems can be used by ys to'recog- 
nize” who the pictures are pictures of.” 

To be cautious, I will mark with corner guotes 
(““and“”) terms we standardiy apply to comput- 
ers that 1 believe rely on our interpretation of the 
semantics of the action or structures, rather than 
anything that the system itself can be credited with 
understanding or owning. Ihus: image or face 'rec- 
ognition , algorithmic 'decision making” and so on. 
(Perhaps we should even say 'computing' the sum 
of 7 and 13, but that is for another time.) 


3. If the capacity is built into a camera, one might argue that at least 
the camera is computing a mapping between real people and other in- 
formation about them, But the guestion of whether what the system 
associates with the other (represented) information is the person in 
view, or their representation on the cameras digital sensor, is vexed. 
See the discussion of adversarial examples in chapter 6, note 5 (p. 57). 


D2 Learning: Perhaps the most significant property of 
ML systems is that they can be trained. Using Bayes- 
ian and other forms of statistical inference, they are 
capable of what I will call "learning —a holy grail 
of AL, with respect to which the classic first-wave 
model provided neither insight nor capacity. 

Several architectural facts are ctitical to the ca- 
pacity of ML systems to be trained. Ihe complexity 
of the relatively low-level but extremely rich search 
spaces, architecturally manifested in high-dimen- 
sional real-valued vectors, enable them—given 
suftiıcient computational horsepower (see Da, be- 
low)—to use optimization and search strategies 
(especialiy hill-climbing) that would be defeated in 
low-dimensional spaces.* Egually important, at the 
relevant level of abstraction the correlation spaces 
need not be discretely chunked—allowing steady 
incremental transitions between and among states, 
the epistemological opposite of ideas remaining 
“dear and distinct.” 

Metaphorically, we can think of these processes 


4. Ihe higher the dimensionality of the search space—the greater the 
number of independent variables—the less likely it is that hill-climb- 
ing algorithms (strategies that move in the direction that, locally, has 
the steepest upwards slope) will encounter local maxima. 

5. İt was never clear how Cartesian models could accommodate the 
gradual shifting of beliefs or of concept meanings, except by the ex- 
cessively blunt addition or removal of specific discrete facts. Estima- 
ble efforts were made within the GOFAJ assumptions, including the 
tradition of non-monotonic reasoning and belief revision or mainte- 
nance, See, for example, Jon Doyle, “A Truth Maintenance System, 
Artificial Intelligence 12, no. 3 (1979): 231-272; and Peter Gördenfors, 
ed., Belief Revision (Cambridge: Cambridge University Press, 2003). 
But it is fair to say that learning remained an Achilles' heel of first- 
wave Al, 


as moving around continuously in the submarine 
topography depicted in chapter 3's figure 6 (p. 34), 
making much less mysterious their ability to “come 
above water” in terms of what we linguistic observ- 
ers take to be 'discrete conceptual islands.” That 
is not to say that the conceptual/nonconceptual 
boundary is sharp. Whether an outcropping war- 
rants being called an island—whether it reaches 
“conceptual” height—is unlikely to have a determi- 
nate answer, İn traditional philosophy such gues- 
tions would be called vagve, but 1 believe that label 
is almost completely inappropriate. Reality—both 
in the world and in these high-dimensional repre- 
sentations of it—is vastly richer and more detailed 
than can be'effably” captured in the idealized world 
of clear and distinct ideas. (There is nothing vague 
about the submarine topology of figure 6; it merely 
transcends ready conceptual description.) 


D3 Big Data: Once trained, machine learning sys- 
tems can respond to inputs of limited complexity 
(ehough often still substantial; a single image from a 
good digital camera uses megabytes of data). Train- 
ing these systems, however, at least given the pres- 
ent state of the art, reguires vastly more data. Ihis is 
why machine learning is at present a “post Big Data” 
development; training involves algorithms that sort, 
sift, and segment massive amounts of it, culling sta- 
tistical regularitles out of an overwhelming amount 


of detail, 


6. Humans may reguire massive initial training sets, too—the idea 
being that early childhood may be a long training seguence for in- 
fants, in order to set up the initial prior probabilities needed for sub- 
seguent recognition and processing, 


D4 Computational Power; Iraining algorithms can 
reguire phenomenal amounts of computational 
power.” Some systems in current use employ the 
parallel processing capacities of banks of GPUs 
(video cards) —up to thousands at a time, each ca- 
pable of processing thousands of parallel threads at 
gigahertz speeds. 


Ihelast two points are historically significant. As Geoffrey 
Hinton has remarked,* they reflect the substantial truth in 
the (in)famous 1973 Lighthill Report,” which threw cold 
water on the idea that first-wave Al could ever scale up 
to produce genuine intelligence. Given not only the ideas 
on which it was founded, but also the amount of com- 
putational power available at the time, fırst-wave Al was 
indeed doomed. The million-dollar, room-filling comput- 
ers on which GOFAIJ was developed" had less than a mil- 
lionth the processing power of contemporary cellphones; 
banks of current-day parallel processing video cards can 
extend that power by yet additional factors of hundreds 
or thousands, 

But Al moves forward. Using different ideas, masses 
of collected data, and tradically improved hardware, the 
results of machine learning are genuinely impressive. Re- 
current networks, deep reinforcement networks, and other 
architectures are being developed to deal with time, to 
push feedback from later stages in a process back to ear- 
lier ones, and so on. New accomplishments are published 


7. Ihis is especialiy true of those in use at the time of this writing, 

8. Geoffrey Hinton, personal communication, 2018. 

9. James Lighehill, “Artificial Intelligence: A General Survey” in Artifi- 
cial Intelligence: A Paper Symposium, Science Research Council, 1973. 
10, Primarily Digital Eguipment Corporation PDP6s and PDPros. 


almost  daily—transforming machine 'translation ,” 
"reading" X-rays, filling in deleted portions of images, 
and such. Certainly Al researchers are more excited and 
optimistic than they have been in 50 years; it is not just 
the press that is heady. I too agree that the developments 
portend profound changes to the nature of society and our 
self-understanding. 

Does that mean that we have fıgured out what it is to 
think? I think not. 


ıı. Google translation is especially impressive when the languages are 
similar linguistically and capable of similar registrations; increasingiy 
less so as these similarities fall away. 


6 — Assessment 


How well do machine learning and second-wave Al deal 
with the four critigues of GOFAI? 


6a - Neurological 

Ihe neurological critigue may be sufficientiy addressed. 
Inspired by what we know about low-level neural organi- 
zation, contemporary machine learning architectures do to 
some extent mimic the brain. As suggested in chapter 3, 
though, the importance of that architectural similarity is 
not entirely clear. For one thing, it would be premature to 
assume that what matters for our brains epistemic power 
is our general neural confıguration— shared by all higher 
mammals. Ihat is all that current architectures mimic. 
Second, as programmers know, it is easy enough to imple- 
ment one kind of architecture on top of another, albeit 
sometimes at a significant performance cost. While it is 
unlikely that evolution would have engaged in architectur- 
al mapping at the level of low-level neural circuits, the ver- 
dict is still out for the higher level forms of reasoning that 
are distinctive of our (so far) uniguely human capacities, 
which seem unlikely to be a simple conseguence of general 
confıguration. Third, there is the fact that no one knows 
whether working in the way our brains do is the only or 
even best route to general intelligence.' 


1. İf, as may turn out to be the case, the ontological structure of the 
world is such as effectively to reguire massively parallel networks for 


Nevertheless, the parallelism of ML architectures, and 
perhaps their statistical abilities to deal with networks of 
probabilities, will likely be of lasting significance, perhaps 
particularly at the perceptual level (though it is ironic that 
ML systems currently take so much computational pow- 
er to train, given that anything neurally realistic must be 
slow). 


6b - Perceptual 

Regarding the perceptual critigue, ML systems again seem 
closer to the mark. Their impressive performance on face 
"recognition' tasks, for example, is telling. As usual, more- 
over, the lesson is as much ontological as architectural, Ac- 
cording to our best current understanding, it turns out that 
what makes faces distinctive are high numbers of complex 
weakly cotrelated variances across their expanse—the 
very sorts of feature that ML architectures are suited to 
exploit—rather than the presence of a few gross charac- 
teristics.” In general, visual 'recognition' of scenes, and al- 
lied tasks such as 'reading' X-rays, voice identification, and 
so on, are the types of task at which ML most evidentiy 


interpretation, then that would be a reason for Als to have such ar- 
chitectures—but the logical structure of such an outcome would be 
that Als and brains would be similar for the same reason, not per se 
that Als need to mimic brains. 

2, See the discussion of Feldmans “100-step rule” in chapter 3, note 3 
(pp. 23-24). 

3. One might wonder whether there may not be a small number of 
characteristics in terms of which people can be recognized and iden- 
tifred, characteristics that we theorists have not yet discovered— or 
even that recognizing those characteristics is exactly what successful 
ML systems do. But if it takes a ML-type architecture to extract those 
characteristics from records of the physical light profiles reflected 
from faces, the point may be moot. 


a 


excels.* Successes in these realms are a substantial part of 
what has fueled contemporary excitement about the power 
of second-wave Al. 

Still, it would again be premature to extract sweeping 
conclusions from these results on their own. It is sobering, 
to take just one example, that many of todays ML image 
recognition algorithms can be defeated by what to humans 
seem trivial changes in the images with which they are pre- 
sented.” In the next chapter I will suggest one reason why 
this might be so. 


6c - Ontological 
Regarding ontology, the issue is trickier (I will consider 


4. Pace the standard caveat that calling such accomplishments “recog- 
nition” is a possibly culpable shorthand for saying that they are able to 
learn and repeat mappings between images of individual entities and 
some other computational structure associated with them. 

5. For examples of such “adversarial” examples, see, for example, 
Acthalye et al.,“Synthesizing Robust Adversarial Examples,' Proceed- 
ings of the 35th International Conference on Machine Learning (Stock- 
holm, Sweden, PMLR 80, 2018). 

As argued throughout the text, the fact that these adversarial ex- 
amples work is evidence that current-generation systems may not in 
fact doing be perception at all, in the sense of dealing with an image 
of a distal situation—but rather something closer to mere image pat- 
tern matching, which we interpret as perception. Ihat is, at best it is 
"perception | 
6. As has been clear since the beginning, I use the term “ontology” 
in its classical sense of being the branch of metaphysics concerned 
with the nature of reality and being—that is, as a rough synonyms 
for “what there is in the world? (1 defer discussion of the relation be- 
tween ontics and ontology, technically its study, to another occasion.) 
As in so many other cases, the term 'ontology” has unfortunately 
come to be redefined, in contemporary computational contexts, to 
refer to structures that represent reality: classes, data structural types, 
concepts, etc.—icensing such otherwise inscrutable constructions as 


the epistemological critigue presently). Machine learning 
is not committed to any particular ontological story, so 
only indirect conclusions can be drawn. As already men- 
tioned, moreover, discussions of second-wave Al typicaliy 
focus on “uninterpreted” internal confıgurations (patterns 
of weights, activation strengths, transformations, etc.), 
obscuring whatever assumptions are being made about 
the nature of the world those confıgurations represent— 
about which different researchers, moreover, undoubtediy 
have different views. In addition, the relentless pace of 
ongoing research makes ML an unstable target of anal- 
ysis. Still, it is not too early to say that the successes of 
second-wave Al provide evidential support both for the 
ontological critigue itself and for the metaphysical view 
introduced in chapter 3. 

Itisin the realms of perception and action that machine 
learning most obviously overcomes the limits of formal on- 
tology, and attends to the “subconceptual” terrain suggested 
in chapter 3's figure 6 (p. 34). When fed with data obtained 
directly from low-level sensors—visual pixels, haptic sig- 
nals, and so on— ML systems have vastly improved on the 
levels achieved in GOFAJ, even achieving levels competi- 
tive with human petformance. This is enabled by a number 
of factors, including continuous patterns of weights, which 
allow for incremental adjustment and training, and suff- 
ciently high dimensionality to'encode” all kinds of subtlety 


and nuance.” The resulting systems are particularly im- 


“creating an ontology, and'ontological engineering." It is the world it- 
self I am interested in here; I defer guestions about its representation 
to considerations of epistemology. 

7.It can be argued that the values of discrete pixels impose a “formal” 
grid on the array of impinging radiation, and therefore that the data 
are not genuinely continuous, but even if that were relevant a stream 
of values does not per se implicate an object, and the value of any such 


pressive in not being defeated by intermediate cases, in ac- 
commodating noisy data, and in being robust in the face of 
ambiguity (as well, of course, as being able to be trained) — 
all of which capacities rely on the fact that they do not have 
to categorize and discretize their inputs at the outset. 

In part, the accomplishments of contemporary systems 
stem not just from their being oriented toward a wealth 
of subconceptual detail, but also from their ability to store 
and work with it, rather than merely attending to it when 
initially presented— especially to integrate large amounts 
of information extracted from it into adjusted weights in 
the activation networks. 'Ihis ability to ingest vast amounts 
of detail gives them a leg up on some perceptual tasks, and 
is critical in allowing them to move beyond what is hu- 
many possible. 

Our visual systems, too, seem capable of processing 
staggering amounts of low-level visual data when imme- 
diately presented with it, but it is less easy to imagine that 
we can store anything close to all of it, once effective cou- 
pling with the input is removed. Yet saying anything defi- 
nite about human information retention is diffıcult, given 
out cutrent ignorance about exactly how the brain works, 
Artists and visualiy-oriented people display stunning rec- 
ognition ability for faces and individual scenes not recentiy 
encountered, for example—a facility that suggests not only 
nonconceptual but also informationally dense memories 
or predictive fabrics of expectation. 

Still, in a point of considerable significance not only to 
Al but also to cognitive science and philosophy, it is com- 
monly assumed that the role of perception, in the human 
case, is to take in the vast complexity of the perceptual 


readout is highly context sensitive, affected by incident illamination, 
the cameras position and orientation, and numerous other factors, 


input and to output a 'conceptual parse” of it—a concep- 
tual parse of what is “out there,' that is—arrayed in terms 
of familiar (and effable) ontological categories, no longer 
“burdened” by the wealth of detail that led to it. Once the 
perceptual input is categorized, that is, it is assumed, on 
the GOFAIT model in particular and in many (especialiy 
analytic) philosophical models of mind, that an intelli- 
gent system can discard the detail that led to that abstractive 
categorization, and that reasoning or rationality from that 
point forward can operate putely in terms of the catego- 
res (LE; purely in terms of sentences, propositions, or data 
structures categorically framed). Ihis assumption fits into 
a general story that human categorization is at least in part 
a technigue for avoiding information overload—abstrac- 
tion in order not to swamp the capacity of the brain. It is 
also, as we will see, the idea that underlies the Cartesian 
desire for'clear and distinct” ideas. 

'Ihe success of second-wave Al suggests that reasoning 
need not work that way—and that it may not even work 
that way in humans, 

One way to depart from the classical'discard the details” 
approach to classification is to avoid categories altogether. 
While we humans may classify other drivers as cautious, 
reckless, good, and impatient, for example, driverless cars 
may eschew discrete categories and chunking entirely, in 
favor of tracking the observed behavior of every single car 


8. As technology advances, one might imagine that computational 
memory will be less easily swamped than our own. Time will teli, 
though it is sobering that even at today$ state of the art, storing high- 
resolution video streams of all video cameras in operation remains 
challenging. Still, prospects of incredibly dense computational stor- 
age (e.g, DNA-based) will allow us to store many orders of magni- 
tude more information that we do at present. How that will affect our 
environment and the fate of Al systems no one yet knows, 


ever encountered, with that data then uploaded and shared 
online—participating in the collective development of a 
profile of every car and driver far in excess of anything hu- 
manly or conceptually graspable. Or to consider a differ- 
ent case, BluelDot, a Toronto startup,” collects worldwide 
travel itineraries, including billions of airline itineraries a 
year, to aid tracking and predicting the global spread of in- 
fectious diseases. Whereas traditional epidemiology tests 
on discrete categories or characteristics (middle-aged man, 
cancer sufferer, abuse sutvivot, etc.), no technical teason 
prevents a ML system from tracking all individual medical 
records, and dealing solely with vastly dimensioned vectors 
of real numbers, without any evident need to compart- 
mentalize the data. Ihe promise of “personalized” medi- 
cine, medical records with individuals' DNA seguences, 
and so on, may similarly"get in underneath the categories,' 
to impressive effect.“ 

Even if a network does 'classify' something, more- 
over —as a person, intersection, political dispute, war zone, 
whatever—it need not do so classically. Nothing in these 
architectures reguires that, in “selecting” some conceptual 
category, the system must discard the trove of detail that 


9. https://bluedot.global 

10. Ihere will be analytic challenges in how we understand such sys- 
tems. Whereas a traditional diagnosis might be phrased as “you have 
a 5290 chance of having melanoma” on a freguentist interpretation 
(i.e., because 5296 of the people in this or that group that you are now 
identifled with have developed melanoma), that may not be an appro- 
priate way to cast a ML systems conclusion. It is not that probabilities 
will not pertain, just because a system is not dealing with groups; in 
fact most ML architectures are defined in terms of probabilities. The 
probabilistic diagnoses they come up with, however (or that we de- 
rive from their calculations) may reguire interpretation in something 
more like an epistemic measure of certainty:“Iam 5296 confident that 
you in particular have melanoma, based on what I know. 


led to that resule— detail that may provide information 
about the warrant for the classification, inflect it with inef- 
fable shading and modulation, relate it to other concepts 
(neighboring islands), and so forth. In fact the very claim 
that the system has classifled something may merely be a 
statement on our part, as external obsetvers, that the pat- 
terns of weights and activations are “within the region” as- 
sociated with the discrete labels “person”, “war zone,” and 
so on, Ünless the system is reguired to make a hard-edged 
choice” among discrete alternatives, that is—such as to 
output a discrete token or word, corresponding to our hu- 
man categories— even the distinction between whether or 
not a system has classified something need not be sharp. 

Moreover, the success of ML systems in cases of simple 
reasoning shows that retaining and working with the sta- 
tistical details and correlations derived from a “submarine” 
ontological perspective can convey substantial inferential 
power (making the reasoning for that reason at least par- 
tially nonconceptual). It is exactly such capacities that em- 
power the widely touted era of Big Data. What is transfor- 
mative about the present age is not just that we have access 
to mountains of conceptualiy represented facts, but that 
we have developed computer systems with predictive and 
analytic power enabled by their ability to track correlations 
and identify patterns in massive statistical detail, without 
having to force-fıt those patterns of relation into a small 
number of conceptual forms. 


1. Technically this should be "choice", but it would be pedantic to 
mark every possible instance of the distinction. Plus, we all employ 
what Dennett would call an “intentional stance” in our characteriza- 
tions of computers (Dennett, The İntentional Stance, Cambridge, 
MA: MIT Press, 1987). 1 will mark just those cases where it is most 
important that we resist the tendeney to attribute more capacity to 
the system than is warranted. 


n 


No one of these facts about the successes of second- 
wave Al is ontologically determinative; none provides 
invincible evidence of how the world is. But the more 
successful these systems grow, the more compelling the 
argument that the “coarse-graining” involved in interpret- 
ing the world through articulated concepts and discrete 
objects (i.e,, interpreting it through the lens of formal on- 
tology) is an information reduction strategy for purposes 
of calculation, reasoning, or verbal communication, rather 
than corresponding to any definite prior in-the-world 
discretization. Yes, we may talk as if the world were on- 
tologically discrete; and yes, too, we may believe that we 
think that way. It seems increasingiy likely, however, that 
such intuitions” reflect the discrete, combinatorial nature 
of language and articulation more than any underlying on- 
tological facts, and also more than the patterns of tacit and 
intuitive thinking on which out articulations depends.” 


12, A natural suggestion, from ML architectures, is that intuitions are 
(distributed) patterns of weights or activations formed in the high- 
dimensional networked representations of the richly interconnected 
submarine topologies of the world that underlie our concepts. The 
common diffıculty we have'expressing” them may reflect the fact that 
words and discrete concepts are excessively bulky, insensitive tools 
with which to capture their ineffable modulation and subtlety. 

13. Ihese lessons are increasingly recognized within Al itself, Rich 
Sutton, a founder of computational reinforcement learning and lead- 
ing ML scientist, recently put it this way: “We have to learn the bitter 
lesson that building in how we think we think does not work in the 
long run. ... (I)Jhe actual contents of minds are tremendousiy, irre- 
deemabiy complex; we should stop trying to fınd simple ways to think 
about the contents of minds, such as simple ways to think about 
space, objects, multiple agents, or symmetries. All these are part of the 
arbitrary, intrinsically-complex, outside world. ... (IM)heir complexity 
is endless” (Rich Sutton, “The Bitter Lesson,” http://www.incomple- 
teideas.net/Incldeas/BitterLesson.html, emphases added.) 


In the face of second-wave Al, in sum, Descartess idea 
that understanding must be grounded on “clear and dis- 
tinct ideas” seems exactly backwards. 'Ihe successes of ML 
architectures suggest that a vastly rich and likely ineffable 
web of statistical relatedness weaves the world together 
into an integrated “subconceptual” whole.'* That is the 
world with which intelligence must come to grips. 


* * * 


As always, care must be exercised when drawing such on- 
tological lessons from the current state of the art. 

First, we typically feed ML algorithms with data that are 
already processed, and to that extent “postconceptual”: sex 
or gender selected from a short list of discrete possibilities, 
experience measured as various forms of unidimensional 
scalar, videos of traffhıc at what we humans classify as “inter- 
sections,” even light intensity coming from some preclas- 
sifled direction, and so on. Even if it looks on the surface 
as if the ML system is dealing with the world at a pre- or 
nonconceptual level, that is, there are many ways in which 
human conceptualization can sneak it—in ways unaccom- 
panied by darificatory subconceptual detail. 

Çuestions need to be asked about the origin, appropri- 
ateness, bias, and so forth, of all such groupings and factor- 
ings—indeed, about the full range of data sets on which 
such systems are trained. If a ML system is given pixel-level 
detail about images of people or plants, it may not need 
to make a binary 'decision' about whether some plant is 
a bush or a tree, or whether a person is brown or white. 
But if it is being trained on databases of images that have 
been categorically tagged by human observers, any subcat- 
egorical subtlety and traces of prejudicial nuance will in all 


14. See my “Ihe Nonconceptual World,” unpublished manuscript. 


likelihood have been lost, and the system is liable to fall, 
without "knowing' it, into derivative patterns of bias and 
prejudice. If fed data from 'Iwitter, Facebook, and similar 
sources, for example, ML systems famously inherit and 
reproduce patterns of racism, public shaming, false news, 
and the like—all without, as it were, batting an eyelash. 

We humans are of course affected by the discourses in 
which we participate, too, but one can at least hope that 
humans will bring a critical or skeptical attitude to such 
sources in a way that ML systems are as yet unable to do. 
Such reflective critical skills are exactly of the sort that I 
argue cannot arise from ever-more-sophisticated second- 
wave technigues. Instead, they will reguire what I will call 
full jadgment.” 

A second reason for caution in interpreting the suc- 
cess of second-wave Al stems from the fact ML systems 
are increasingly dedicated to sorting inputs into categories 
of manifest human origin and utility. To the extent that 
they are designed to mesh with our categories, even if they 
retain subconceptual detail, they will nevertheless thereby 
adopt, and be affected by, those categories' interests, util- 
ity, and bias. And if the outputs are discrete categorical 


15. As mentioned in the introduction, it is possible that if second- 
wave Al were used as a basis for a class of synthetic creatures (per- 
haps along the lines of Sony's Aibos) that were themselves able to 
evolve, then over a long period of time such creatures might (as we 
did) eventually develop full-blooded tationality and jadgment. See 
the discussion of 'creatures” in chapter 10. The point is just that such 
a capacity would depend on their forming cultures and community, 
making and living by commitments, being governed by norms, going 
to bat for the truth, etc. They would not be capable of jadgment, if 
indeed they ever reached that stage, merely in virtue of supervening 
on second-wave Al technigues. An explanation of their thereby de- 
veloped normative capacities, therefore, would necessarily advert to 
more than their being just machine learning or second-wave systems, 


dassifıcations, as suggested above, and if we design the sys- 
tems based on our classical myths about the nature of clas- 
sification, the abundance of detail on which they rest, and 
thus any subtleties about the origins and appropriateness 
of such categorizations, are likely to be lost. 

A third caution concerns a topic of some contemporary 
urgency: increasing calls for ML systems to 'explain” their 
actions may prove to be curiously petverse. Ihe abilities 
that the systems are being pressed to 'explain' may be 
powerful exactly because they do not arise from the use of 
the very concepts in terms of which their users now want 
their actions accounted. Ihe pressure to develop “self-ex- 
plaining” or “interpretable” neural networks, that is, may 
inadvertently decrease their performance, and drive them 
toward unwarranted reliance on binary or discrete cat- 
egories, toward implicit or even explicit reliance on formal 
ontology— may drive them, that is, back toward the episte- 
mological and ontological inadeguacies of GOFAI, 


What should we make of all this? While the state of cur- 
rent research is messy, and clear-cut conclusions diffıcult 
to draw, I believe three ontological morals can be drawn. 


ı. Ihe dassical assumption of a discrete, object-based 
“formal” ontology is not a preteguisite of machine 
learning and other second-wave Al technigues. On 
the contrary, the success of ML systems, particular- 
İy on perceptual tasks, suggests a different picture: 
that the world is a plenum of unbelievable richness, 
and that the familiar ontological world of objects, 
properties, and relations, represented in articulated 
conceptual representations, is vety likely “the world 
taken at a relatively high level of abstraction,' rather 
than the way that the world is, 


2. Much of ML power stems from its ability to track 
correlations and make predictions “underneath” 
(i.e,, in terms of vastly more detail than is captured 
in) the classificatory level in terms of which such 
high-level ontology and conceptual registration is 
framed. 

3. İhefactthat ML systems are increasingiy being tar- 
geted toward domains that have been ontologicaliy 
prepared by—and targeted for—humans, typically 
in conceptualiy structured ways, inevitably leads 
these systems to inherit both the powers and limi- 
tations of human approaches, without any critical 
faculties in terms of which to guestion them. It is 
these factors that are giving rise to the widely dis- 
cussed (but inappropriately described) phenom- 
enon of algorithmic bias”” 


Years ago, 7 as noted in chapter 3, I outlined a picture of 
the world in which objects, properties, and other ontologi- 
cal furniture of the world were recognized as the results 
of registrational practices, rather than being the pregiven 
structure of the world. Ihe picture is useful in terms of 
which to understand both the failures of GOFATJ and the 
successes of machine learning. It depicts a world of stu- 
pefying detail and complexity, which epistemic agents reg- 
ister —fınd intelligible, conceptualize and categorize—in 


16, It is the data—not only its form and its content, but also attendant 
factors about its selection, use, etc.—that is the primary locus of bias 
in machine learning results. The algorithms that run over the data 
are undoubtediy not innocent—reguiring data sets to be formed in 
particular ways, etc. But most examples of bias cited in the press and 
literature are due more to skewed data than to culpable algorithm. 
We need critical assessments that properly tease apart the respective 
contributions of these two dimensions of ML architectures. 


17. On the Origin of Objects, 1996. 


order to be able to speak and think about it, act and con- 
duct their projects, and so on. Most importantly, the view 
was developed from the ground up to take seriously the 
fact that most of the world—indeed, the world as world— 
outstrips the reach of effective access, necessitating discon- 
nected, semantic representations that, for reasons of com- 
plexity, necessarily abstract away from most of its suffusing 
detail, It leads to a picture of conceptual prowess as most 
relevant to the intelligibility of relatively more distal situa- 
tions, and nonconceptual skills as particularly appropriate 
for the suffusing detail of the immediately nearby. As 1 put 


e 18 
itin another context: 


“I sometimes think of objects, properties, and rela- 
tions (i.e., conceptual, material ontology) as the long- 
distance trucks and interstate highway systems of 
intentional, normative life. Ihey are undeniabiy es- 
sential to the overall integration of lifes practices— 
critical, given fınite resources, for us to integrate the 
vast and open-ended terrain of experience into a sin- 
gle, cohesive, objective world. But the cost of packag- 
ing up objects for portability and long-distance travel 
is that they are thereby insulated from the extraordi- 
narily fine-grained richness of particular, indigenous 
life—insulated from the ineffable richness of the very 
lives they sustain.” 


Both the successes and limitations of fırst- and second- 
wave Al make eminent sense in terms of this picture— 
predictable characteristics of architectures beginning to 
extract the rich but radically simplifying registrations fun- 
damental to perception and cognition. 


18.“Ihe Nonconceptual World," unpublished manuscript. 


6d - Epistemological 
What then about epistemology— subject matter of the re- 
maining GOFAIJ critigue? 

Here the rubber fınally meets the road. Iwo major is- 
sues, to be addressed in the next chapter, stand in the way 
of Als reaching anything that can truly be called thinking. 
Both have to do with what is involved in holding think- 
ing and intelligence accountable to the fabulously rich 
and messy world we inhabit. One, relatively straightfor- 
ward, involves reconciling the first- and second-wave ap- 
proaches— taking advantage of their respective strengths, 
and moving beyond at least some of their limitations. This 
integrative goal is starting to be recognized, and to be sug- 
gested as a necessary ingredient for third-wave Al, 

Ihe other challenge is more profound. I do not believe 
that any current technigues, including any yet envisaged as 
a subject matter of Al research, even recognize the impor- 
tance of this second issue, let alone have any idea of what 
would be involved in addressing it. Explaining it will take 
us into realms of existential commitment and strategies for 


dealing with the world as world. 


7 — Epistemological Challenges 


Start with the first and easier goal, of integrating the mer- 
its of both fırst- and second-wave Al, 

One of GOFAİS strengths was its ability to deal with 
articulated reasoning—inferences involving (some- 
times long) chains of structured propositions involving im- 
plications, negatives, guantifıcation, hypotheticals, and so 
on.“Since Randy and Pat are in Tokyo for the Paralympics, 
they wont be here for dinner on Saturday" for example. 
Or: “179 of Londons population have parents who speak 
different mother tongues. Or even: “One reason why pub- 
lic support for higher education is proportionalIy lower in 
Canada than in the U.S. is that social healthcare, unassail- 
able in Canada but not in the U.S,, takes up a large frac- 
tion of the public purse, and Canadian voters are reluctant 
to devote much more money to other social programs—a 
category of which education is viewed as an instance” An 
illustrative list of the articulated capacities that even ear- 
İy Al systems were designed to deal with is given in the 
sidebar on the next page. Ihey remain critical to general 
intelligence." 


1. İhe articulative capacities listed in the sidebar are interdependent, 
both syntactically and semantically, Some of the relations among 
them are theorized in cognitive science and philosophy under notions 
of productivity (the fact that cognitive production and comprehension 
are unbounded), systematicity (the fact that the meanings of whole 
sentences and whole thoughts are systematically related to the mean- 
ings of the words or tokens they are made up of), and compositionality 


Characteristics of Articulated Reasoning 
Some forms of conceptual structure that were common features of knowl- 
edge representation and models or reasoning in the GOFAJ mold. 


ı. Identity and nonidentity (“Tully is Cicero,”"the baker is not 
my aunt Hilda”) 

2. Guantification (“every Canadian owns a tugue,' “there is a 
rattlesnake in the grass over there”) 

3. Variables (“marriages in which one parent of each partner im- 
migrated from the same country”) 

4. Logical operators (and, not/negation, implies, etc.: “no one 
who is both a cutler and a classical violinist likes garlic”) 

5. Sets (the president, vice-president, and treasurer,' “all bono- 
bos in captivity”) 

6. Opacity and intensional contexts (“she said that France has a 
king,'“he believes that 7 is rational”) 

7. Categories and subcategories (“angel investors,' “evetyone 
who was invited”) 

8. Possibility and necessity (“she might have transferred from 
Swarthmore”) 

o. Default reasoning (“unless otherwise noted, the tide must be 
considered in the design of all saltwater harbors”) 


(che fact that the meaning of a complex sentence or thought is deter- 
mined by its grammatical structure and by the meanings of its con- 
stituents). he significance of these combinatoric relations for human 
cognition is bluntly suimmarized by Jerry Fodor, but the point would 
generalize to any computational system aiming at achieve general 
intelligence: “Human cognition exhibits a complex of closely related 
properties—incuding systematicity, productivity and compositional- 
ity—which a theory of cognitive architecture ignores at its peril. If 
you are stuck with a theory that denies that cognition has these prop- 
erties, you are dead and gone.” Jerry Fodor, “Connectionism and the 
Problem of Systematicity (Continued): Why Smolensky's Solution 
Still Doesn't Work,” Cognition 62, no. 1 (1997): 109—119. 


If GOFAT could deal with such logical intricacies, 
why did it fail? Because, as has been argued throughout, 
the ontology in terms of which it was formulated was all 
wrong, First-wave Al had no resources to ground its ab- 
stractions and conceptual symbolizations in the tichness 
of the world, giving them a tendency to float free of real- 
ity—tchreatening them to devolve, as it were, to mix a little 
Derrida and Shakespeare, into an endless play of signifters, 
signifying nothing. 

Still, the internal structure of articulated reasoning is 
critically important, and must be a part of any genuine 
model of intelligence. Our current best understanding of it 
has been worked out in the context of logic, complete with 
its presupposition of formal ontology, clear and distinct 
concepts, and so on. Ihink about such paradigmatic logical 
expressions as “VxJF(x)5G(x)7” or (p>5b1Wv-p)” These 
formulae are normally interpreted with reference to deter- 
minate semantic and ontological facts about the meanings 
of their constituents. Any x is assumed to be dearly indi- 
viduated and either F or not F, and G or not G, without 
ambiguity or matter of degree. Similarly for ©, W, and so on. 

It was argued in the last chapter that perception, clas- 
sification, nonconceptual reasoning, and even the sorts of 
reasoning employed in analyses of Big Data, do not reguire 
that degree of ontological clarity—in fact, that their power, 
as demonstrated in ML systems, arguably derives exactly 
from not assuming it. Per se, that does not address logi- 
cal complexities of the present sort, though it is clear that 
human intelligence does not teguire perfectiy “clean” cat- 
egories in order to deploy commonsense versions of such 
logical relations. (Ihe longer the chains of reasoning grow, 
the more likely conclusions will fall apart— grow false or 
unreliable—in propottion to the extent that the relevant 


objects and properties are not homogenous and sharp- 
edged.) Support for articulated forms of reasoning is one 
of conceptual classification's great powers—but how clean 
or 'discrete” the categories must be in order for it to be use- 
fully applicable is far from well understood. It is also clear 
that some degree of logical reasoning can apply to noncon- 
ceptual content (that red sofa will not go well with the 
wallpaper in our living room,'"write here on the board, not 
over there/ etc. ). 

'Ihese considerations raise a suggestion for Al, prefig- 
ured in the last chapter: to figure out what would be re- 
guired to construct an architecture able to support ML's 
extremely rich, high-dimensional representational vectors 
as elements in complex patterns of articulated reasoning. 
Fuzzy logic might be considered an early stab in that direc- 
tion, but it was restricted to single, real-valued truth val- 
ues, and was “higher-order discrete” in the sense that any 
given “fuzzy value” was exactly (rather than “roughly”) what 
it was represented as being—that is, some exact real num- 
ber. Rather, what is being suggested is to develop systems 
that integrate ML5 ontologically rich and nonconceptualiy 
presumptive representations with the patterns of articulat- 
ed reasoning paradigmatic of first-wave Al—not by glu- 
ing the two capacities together in a “bimodal” system, but 
seamlessly integrating them, so that the nuances, subtle- 
ties, adjustments, and so on, embedded in the underlying 


2. Ihese examples are considered nonconceptual because the sense 
of redness pointed to in the first is likely not redness in general, but 
the particular red of the sofa—a shade that the speaker is unlikely to 
possess adeguate conceptual resources to describe, and because the 
patches of wall signifed by “here” and “there,' in the second, are un- 
likely to be well-defined regions with determinate boundaries. 

3. John Haugeland, “Analog and Analog,' Philosophical Topics 12, no, 
1 (1981): 213—225. 


subconceptual webs underlying the concepts being com- 
bined could play a role in establishing the subconceptual 
webs underlying the result, in ways that would give nuance 
and inflection to relatively immediate inference, but allow 
increasingly long chains of articulated reasoning only the 
more distilled, abstracted, and'discretized” the representa- 
tions become, 

'Ihe architectural implications of this proposal are non- 
trivial. In the classical (GOFAT) case, complex hypotheti- 
cals, disjunctions, implications, and so on, can easily involve 
dozens or even hundreds or more conceptually articulated 
constituents, İn current neural architectures, anything that 
the systems "know" may be encoded in weights distributed 
across the entire network; how to make states of the net- 
work into parameters or constituents of compositionally 
structured other states of the network is neither evident 
nor likely straightforward. 

Nevertheless, this goal of compositionally structured 
states backed by reams of subconceptual detail does not 
seem fundamentally at odds with the sorts of problem that 
Al researchers are already tackling—and indications that 
the need for such projects is being recognized.* 


4. See, for example, Gary Marcus, Ihe Algebraic Mind: Integrating 
Connectionism and Cognitive Science (Cambridge, MA: MIT Press, 
2001); the papers in Joe Pater, “Genetative Linguistics and Neural 
Networks at 60: Foundation, Friction, and Fusion,” plus comment 
articles (Language, 95:1, 2019); and Hector Levesgue, Common Sense, 
the Turing Test, and the Oyest for Real AT: Reflections on Natural and 
Artificial Intelligence (Cambridge, MA: MIT Press, 2017). 

One challenge for such projects will be to determine whether help- 
ful insights will emerge in the experience and technigues that have 
been developed in contemporary machine learning systems trained on 
the outputs of human conceptualization, such as Wikipedia articles, 
Twitter feeds, etc. Along with the evident issues of bias, prejudice, 
etc., discussed in the previous chapter, a serious issue that would need 


* * * 


Ihe other epistemological challenge is more profound. 

No matter how otherwise impressive they may be, I be- 
lieve that all existing Al systems, including contemporary 
second-wave systems, do not know what tbey are talking 
about. It is not that we humans cannot interpret their out- 
puts as being about things that matter to us. But there is no 
reason to suppose, and considerable reason to doubt, that 
any system built to date, and any system we have any idea 
how to build, 'knows' the difference between: (1) its own 
(proximal) state, including the states of its representations, 
inputs and outputs; and (ii) the external (distal) state of 
the world that we at least take its states, its representations 
and those inputs and outputs, to represent. And it is those 
external states of affairs that they are talking about (re- 
member claim P2, about semantic interpretation). 

What is reguired in order for a system to know what it 
is talking about? What is it that present day systems lack, 
such that they do not? That is the guestion to which the 
rest of the book slowly develops an answer. I can say now 
that it will at least reguire authenticity, deference, and en- 
gagement in the world in which that which is talked about 


to be addressed, explored in the remainder of this book, is that the 
data sources these systems use as training sets are not held system- 
atically accountable to the highly variable conceptions (registration 
schemes) in terms of which they have been formulated. Including the 
results of any such “data mining” without evaluating the registrational 
practice underlying each and evety one of them (every article, every 
post) would constitute exactly the sort of”gluing together” that would 
defeat the aim of the project. 

I suggested that this fırst integrative goal would be casier to address 
than the second, but at a deeper level doing it properly may depend 
on successful treatment of the second as well, 


exists—and that neither interpretability nor “grounded 
interpretation” will suffce.” What this means will become 
dear, but some examples may help in developing motivat- 
ing intuitions. 

Suppose an automatic X-ray reading system constructs 
what we take to be, and therefore call, a 3D model of the 
lungs.” Call the model o. Does the system know what we 
know: the difference between « and the lungs of which a 
is a model? More generally, what is the computer likely to 
understand about the notion of a model, at all? Even if we 
encoded meta-level information into « stating that it is 
a model—that is, even if we were to add something like 
“MODEL(.)” to its data structures—how would the system 
know that that meant that w was a model of something in 
the outside world, in the way we do? Per se, meta-level in- 
formation does not help; the problem simply teCUtse$.” 

By the same token, it is unlikely that AlphaGo and its 
successors” have any sense of the fact that Go is a game, 


5. The engagement must be enough to secure the reference, which is 
something less than genuine engagement with that referent. I can re- 
fer to something that is outside of my light cone, even though engage- 
ment with the limits of light cone are proscribed. My ability to refer 
to a T meson, or to Mesopotamian culture, or to womens experience 
of sexism, are dependent on the intentional capacities of my culture 
and community; I cannot shoulder such intentional directedness on 
my own. But the “reach” to such referents cannot be so indirect, medi- 
ated, and externally brokered that reference is impossible, in the way 
that it is for Siri, which surely lacks the ability to refer to a pizza par- 
lor in any substantive sense, if in fact it can refer at all, 

6.A3D model“under interpretation,' needless to say—a model of the 
lungs'3D structure, that is,notamodel that is itself three-dimensional, 
7. To know that “mooE(«)” means that o is a model reguires exactly 
what I am saying we as yet have no warrant in assuming—that the 
system is capable of reference and denotation (not merely 'reference' 
and 'denotation”. 


8. Including AlphaGo Zero, 


with an illustrious millennial history, played by experts 
from around the world— or even, for that matter, and more 
pointedİy, that there is a difference between the particular 
game it is playing and the representation of that game in its 
data structures. Siri and Alexa, similarly, though they may 
teli you about restaurants, bathrooms, thunderstorms, 
and operating system updates, do not really know what 
restaurants are, or bathrooms, or thunderstorms—likely 
not even system updates. As I have said since the begin- 
ning, these systems, being computational, are semantically 
interpreted, and so we understand their behavior as being 
about and referring to their represented worlds. But they 
do not understand themselves “under interpretation"—or 
realize that their thoughts and deliberations and utteranc- 
es matter only under interpretation. Ihat is why I guestion 
whether what they do warrants the label “understanding” 
at all, At best it is 'understanding) but the stakes are high 
enough in these Al games that that might be a phrasing we 
do well to avoid entirely. 

Ihis is not to say that the structures and behaviors and 
ingredients of these systems are not interpretable by us; I 
take it that they are, asa condition on their being compu- 
tational (P2). Indeed, that is exactly why Siri and Alexa are 
so useful. It is also not to say that contemporary computer 
systems do not act in the worlds that their representations 
are about. Many of them do, increasingly. Nor, important- 
İy is it to say that their symbols do not have determinate 
interpretations. İhis is not the place to rehearse what is 
wrong with Searles Chinese Room argument, but let me 
just say that, in spite of some superficial similarities, I am 
not talking about formality, or about the legions of argu- 
ments (such as Searle's) that a formal system cannot have 


real semantics, because the interpretations are unanchored, 
and can be reassigned at whim. I would go to court to deny 
that the symbols in present day Al systems are “formal” in 
this sense—that is, in the way in which Searle understood 
that word. Some contemporary systems, in particular, are 
plugged into the world in such a way that their symbols 
are unambiguously grounded (think about transfer paths 
in internet routers, database entries in real-time financial 
accounting systems, email addresses, etc.). I would even 
argue that the semantic interpretation of many noncom- 
putational symbols (e.g., words on signs and in books) 
are grounded by the practices in which they play a role. 
But neither signs nor books understand what their words 
mean, even if they have nonarbitrary interpretations. Rath- 
er, | am talking about something deeper—about whether 
the systems in which these (grounded) symbols play a role 
genuinely ynderstand anything. 

Put it this way. At the outset 1 said that semantics, in 
order to be semantics, must be deferential. At the moment, 
although we may design our systems with deferential se- 
mantics, tbe deference is oyrs, not theirs. If we are going 
to build a system that is itself genuinely intelligent, that 
knows what it is talking about, we have to build one that is 
itself deferential —that itself submits to the world it inhab- 
its, and does not merely behave in ways that accord with 
our human deference. To do that, it will have to know (i) 
that there is a world, (ii) that its representations are about 
that world, and (iü) that it and its representations must 
defer to the world that they represent. 


What then is deference, what is the world, and what is it to 
know that there is a world to defer to in which objects ex- 
ist? Ihose are the guestions to which the rest of the book 
is addressed. But first something preliminary: Why does 
it matter? Why should we care whether our creations re- 
ally are deferential —or, for that matter, whether they are 
genuinely intelligent? Would it not be enough if we could 
merely interpret them 4s if they were intelligent, deferential, 
and the rest? 

No, it would not. If we are to trust their deliberations, 
Al systems need to be genuinely intelligent in order to be 
able to take responsibility for the adeguacy of the abstrac- 
tive ontologies in terms of which they register the world. 
Otherwise we should use them only in situations where we 
are prepared to take epistemic and ontological responsibil- 
ity for every registration scheme, every inferential step, and 
every “piece of data” that they use along the way. 

Ihis is not a hypothetical problem. We are already 
presented with the results of data mining algorithms that 
have not just run over large individual data sets (such as 
census tallies) but surveyed vast collections of data sets, 
where we do not know what normative standards, regis- 
trations schemes, ethical stances, epistemological biases, 
social practices, and political interests have wrought their 
influence across the tapestry. Just as we would not trust an 
uneducated child or insensitive journalist to summarize 
the suicidal tendencies of teenagers at risk in diverse cul- 
tures, so too we should not trust an Al system unless we 
can similarly trust its ability to critically assess the metits, 
humanity, conceptual compatibility, legitimacy of assump- 
tions, and so on, of all of the data sets that it surveys—to 
say nothing of those on which it has been trained. 


8 — Objects 


Some of the deepest thinking on these topics has been 
framed in terms of transcendental and existential philoso- 
phies. For insight we could delve into Kants inguiry into 
the forms of sensibility and understanding as conditions of 
the possibility of knowledge of objects as objects. Or, turn- 
ing to Heidegger, we could consider the existential gues- 
tion of the possibility of being toward the being of entities. 
But we do not need such fancy language here. We can ask 
a simple guestion: 


What must be trye of a system in order for it to register 
an object as an object in the world? 


Again, this is not a guestion about what it would take for 
an Al system to represent (or deal with) something that 
we humans take to be an object in the world—or rather, 
to put the point more simpiy, just register an object as an 
object,' since, in line with the etymology, I will take being an 
object to mean being part of what is objective—that is, being 
part of the world. So the issue can be more compactly de- 
scribed as one of what it would be for an Al system to take 
something to be an object—for it to represent or refer to, 


1. By object, in this guestion, I mean nothing very specific. Except 
that it would lose the etymological connection with “objective,' entity 
would do as well. That which is registered need not even be discrete 
or individual, At issue is what it is—what is reguired—for a system 


to register anything, as such, in the world. See On the Origin of Objects. 


and thereby be deferentially oriented toward, something 
that it takes to be an object, that it takes to be in the world. 

'Ihis is nothing that any computer system yet imagined 
can do. Nor do I believe that machine learning, nor any 
other second-wave Al technologies, nor anything I have 
seen proposed for third-wave Al, sheds light on it. 

But that does not mean we cannot make progress. Sev- 
eral things can be said. Seven, in fact.” 


Standards on Genuine intelligence 


I, ORIENTATION: Ihe system must be oriented to- 
ward that which it represents, not merely oriented 
toward, or involved with, its representation. It must 
be“intentionally directed toward it, as philosophers 
would say. Referring to something is a way of being 
oriented toward it, so long as the reference is genu- 
ine—something I will explore. But reference is only 
one, particularly targeted, form of orientation. To be 
more general —to recognize that life is often mun- 
dane coping, navigation, and involvement in every- 
day projects, not theotetical reflection— we could 
use phenomenological terminology and say that the 
system must comport itself toward the object. 

Surely, one might think, a computer can be ori- 
ented (or comportt itself) toward a simple object, 
such as a USB stick. IfT dick a button that 'telis' 
the computer to “copy the selected file to the USB 


2. This list is nota theory of epistemology, or of existential commit- 
ment. İn fact it is not a theory at all. Ihe seven (nonindependent) 
standards are merely properties that understanding and fuli-blooded 
intelligence reguire— criteria that human adults meet, and that syn- 
thetic creatures should attain before we should dub them intelligent, 
and (more urgentiy) before we cede responsibility to them for tasks 
reguiring genuine intelligence or judement, 


stick in slot A," andifin ordinaty circumstances my 
so dicking causes the computer to do just that, can 
we not say that computer was oriented toward the 
stick? 

No, we cannot. Suppose that, just before the 
command is obeyed, a trickster plucks out the orig- 
inal USB stick and inserts theirs. The problem is 
not just that the computer would copy the file onto 
their stick without knowing the difference; it is that 
it does not have the capacity to distinguish the two 
cases, has no resources with which to comprehend 
the situation as different— cannot, that is, distin- 
guish the description “what is in the drive” from the 
particular object that, at a given instant, satisfies 
that description. 

It follows, in the present terminology, that the 
computer has no capacity to deal with the USB stick 
as an object. If, mid-copy, 1 tewire the write head, so 
that instead of writing to the drive, it sprays the bits 
onto my Facebook page, the computer would again 
be clueless, and not just in fact, but necessarily. * 
And because it would be congenitally cueless, there 
is no warrant for saying that it was ever oriented 
toward the stick, as opposed to (at most) being ori- 
ented toward the representation of the stick, or the 
driver mechanism that interacts with it, 

How could a computer know the difference be- 
tween the stick and a description it satisfies (the 


3. For philosophers: the computer is constitutionally unable to dis- 
tinguish between de dicto and de re interpretations of (the computa- 
tional analog of) “what is in the drive” 

4. Ihe point would remain even if there were a checksum calculation 
upon completion of the write; that too could be stealthily mimicked. 


stick currently in the drive”), since at the moment 
of copying there need be no detectable physical dif- 
ference in its proximal causal envelope between the 
two—and hence no way, at that moment, for the 
computer to detect the difference between the right 
stick and the wrong one? That is exactly what (nor- 
matively governed) representation systems are for: to 
hold systems accountable to, and via a vast network 
of social practices, to enable systems to behave ap- 
propriately toward, that which outstrips immedi- 
ately causal coupling. To assume that a situation is 
exhausted by what is causally proximate at any giv- 
en moment (to pledge a priori allegiance to blanket 
mechanism, that is) is exactly to be blind to repre- 
sentation, semantics, intentionality, and normativ- 
iy —blind to being oriented toward the world.” 


2. APPEARANCE VS. REALITY: What does it take to 
be oriented toward something? At a minimum, it 
means that the system must be able to distinguish 
the object from a representation of it—in order to 
be deferential toward the former and not toward 
the latter. That is, it has to be able to distinguish 
appearance from reality.* As already noted, it does 
not suffıce to employ guotation or meta-level data 


5. See the discussion of blanket mechanism in chapter 1 (pp. 3—4). 

6. Some philosophers may feel that any distinction between appear- 
ance and reality reguires a separation between the mind and a mind- 
independent object or a mind-independent world—contrary to the 
constructivist sensibilities endorsed throughout. I believe this way of 
framing the reguirements for realism is vastly too strong. Ihere isa 
huge and highly textured territory between indistinguishability and 
independence—a tetritory inhabited, in my view, by all of ontology, 
that is, by essentially everything. 


structures in such a way that, as outsiders, we can 
take some of its representations (machinations, 
behavior, deliberations, whatever) to be about the 
world and others to be about its representations. 
Ihe system must recognize (not just 'recognize") 
that the object is different from its representation 
of it. To take an object as an object rather than an 
"object, that is—to refer to it as an object—re- 
guires “taking it to be out there.” And since in gen- 
eral the object (targeted by its representations) will 
be beyond effective reach, the system must know 
that that toward which it is oriented will often be 
distal— outside the realm of effective connection, 
outside its immediate causal envelope (once again 
challenging the adeguacy of causalist scientific ac- 
counts). Intelligence, that is, must achieve what 1 
dub a" Robert Browning” criterion: to know an object 
as an object is to know that it exceeds your grasp. 


Note again that the distal (noneffectively available) 
nature of semantic content so obviously applies to 
consciousness that we rarely notice how astonish- 
ing it is. What you have “in mind” (a friend, an im- 
pending exam, a truck coming round the corner) 
is not your interior, proximal mental states or pro- 
cesses that represent those phenomena, but the ex- 
terior, distal phenomena themselves. Moteover, be- 
cause even objects to which you may seem causaliy 
connected, must, in order to be objects, have a past 
and future, both of which are beyond effective reach 
(physics prohibits direct causal connection with ei- 
ther past or future), even objects that are “present” 
transcend, as objects, that which is locally available. 


We saw this divergence in the case of logic: a 
separation between how the system works (causally, 
mechanically) and what it is doing (semanticaliy, 
intentionally, “under interpretation”). Ihe present 
point is that, in order to be genuinely intelligent—in 
order, asI will soon say, to be capable of jadgment— 
a system must (i) “know” the difference between the 
two, and (ii) through use of resources provided by 
the former, be oriented toward the latter. 


3. STAKES: Not only must the system be able to dis- 
tinguish appearance from reality; in order to refer 
to or be oriented toward an object, the system must 
defer to that object. To put it in Searles phras- 
ing,” the system must know that when “word” and 
“world” part company, the world wins.” Else truth 
would be sactificed. In order to stand in a noneffec- 
tive semantic relation to the distal world and to take 
there to be an object out there—to take an object as 
an object, for there to be reference at all —chere must 
be stakes, norms, things that matter, 

As Haugeland emphasizes, a system that is def- 
erentialiy oriented toward the world will go to bat 
for its references being external, will be existentialIy 
committed and engaged. We need to know what 
this means, and to determine whether systems are, 
before we will be in a position to claim that any sys- 
tems of our own devising are genuinely intelligent. 


7. John Searle, Speech Acts: An Essay in the Philosophy of Language 
(Cambridge: Cambridge University Press, 1969). 

8. Some may argue that one never has cognitive access to the world 
itself, only to representations of it, and therefore that there is no way 
for the world itself to win. I disagree with the premise, but even if it 


4. 


LEGIBILITY: İt is a precondition on a systems tak- 
ing something to be something, distinguishing 
appearance or representation from reality, taking 
things to matter, that it fınd the object intelligible— 
or perhaps we could say legible—in the world. 
What is it to be intelligible or legible? 'To be a 
patch or parcel of the world that is registerable with- 
in an accountable registrational scheme (more on 
this below)— that is, to be ontologically sound. Un- 
packing what that means does not reguire pledging 
allegiance to naive realism. We can avail ourselves 
not only of Kant but of Kuhn, Haugeland, social 
construction, cultural anthropology, and a spate of 
other contemporary resources. OntologicalIy, to be 
something—an electron, a faux pas, an on-ramp, 
a committee chair—is to participate in a consti- 
tuted domain of regularities, rules, practices, con- 
fıgurations of reality in which we, as the epistemic 
knowers who are registering the world in whatever 
ways are appropriate to the constituted domain, 
participate, and to which we are committed. To be 
an electron is to fit into the whole physical weave of 
existence that electrons inhabit.'To be a move or en- 
tity in a game, to use Haugelands favorite domain 
(a knight fork,? a home base, etc.), can only be what 
itis within the context of the constituted regime of 
chess or baseball games. Even scientific or natural 
kinds, on Haugeland's (and perhaps Kuhn's) view, 


have roles to play in constituting regimes. 


were true, the concusion would not follow. 'Ihe accessibility of the 


world does not bear on its status as normative arbiter of discrepancy. 
9. A chess position in which a knight threatens two or more opposing 


pieces simultaneously. 


5. ACTUALITY, POSSIBILITY, IMPOSSIBILITY: İf 
there is to be a significant distinction between "get- 
ting an entity right” and failing to do so, there must 
be some feasible and nonarbitrary way of telling 
which is which, in particular cases. In fact it reguires 
that the system embrace a three-way distinction, as 
regards any objects it takes to be objects: 


a, What is the case about them 

b. Whatisnot the case about them, but could be the 
case (so that saying that such was the case would 
be false—an error, a mistake, something in need 
of correction) 

c, What could not be the case about them— what is 
conceptually or ontologically impossible, so that 
a statement to that effect can be rejected out- 
right, at pains of the whole intelligibility system 
collapsing (more on this below) 


'Ihe system must be able to distinguish the actual, 
the possible, and the impossible—truth, falsehood, 
and impossibility. * 

Research has given us insight into what exists, 
and has opened up our imaginations to what could 
be, in virtue of being founded on a set of laws which 
also dictate what cannot be the case, that rule out 
what is impossible. Protons cannot be grey, because 
protons are too small for color to apply. Ihe num- 
ber four was not seen on Çucen St. this afternoon, 


10. Haugeland has a more detailed account of this tripartite structure, 
induding an analysis of what he calls the “excluded zone,' in section 
12 (p. 331) of “Truth and Rule-Following,'in Having Thowght (Cam- 
bridge, MA: Harvard University Press, 1998); see also “Truth and 
Finitude,” in John Haugeland, Dasein Discdosed (Cambridge, MA: 
Harvard University Press, 2013). 


because numbers are not occurrent. Star Wars mov- 
ies may suggest that we might someday be able to 
zip across the universe in hyperdrive, but it is not 
going to happen." No algorithm is ever going to ex- 
amine all possible states of the chess board; no con- 
sistent formal system capable of 'stating' its own 
consistency İs going to 'prove' its own consistency; 
and so on, 

More pragmatically, some things that are not 
strictly speaking impossible we take to be impos- 
sible—such as the cup of coffee in front of me spon- 
taneously leaping two inches up into the air (and 
simultaneously growing just enough cooler for en- 
ergy to be preserved). 

Why do we need the impossible, to say nothing 
of the false, in order for there to be import to things 
being correct—and thus for a system to be able to 
distinguish appearance and reality, and thus to be 
able to take something to be an object? Overall, it 
has to do with holding the whole system together, 
not only in order to distinguish ourselves and our 
representations from what we are thinking about 
and representing, but at the same time, to be ex- 
plored in a moment, in order to hold the whole situ- 
ation we are thinking about to account. 

Holding things to account has a pragmatic 
purpose. If things present as impossible—if the 
evidence you are presented with, including that 
delivered by your sensory systems, suggests that 
something impossible has happened—you double 


ıı. Long-distance guantum tunneling would not count as “zipping 
across the universe,' in my book—but the point is that whatever does 
happen will have to accord with physical law. 


down, check everything, teexamine your means of 
discovery, find alternative ways to discover the same 
entities, seek confırmation from other people, and 
so on. İf I were so much as to begin to think that the 
cup of coffee in front of me leaped up two inches, 


Multiple Registrations 

Alİ ways of registering the world are partial, skewed, appropriate in 
some circumstances and inappropriate in others. This foundational 
fact undergirds all knowing, reasoning, and intelligence. Unless a 
system can shoulder responsibility for acting in constant light of it, 
we should not trust its deliverances further than we can trust the 
adeguacy of the registration schemes it employs. 

Ihe point has strong implications for any practice of integrating 
multiple data sources, including data mining and all other uses of 
“Big Data” Genuine intelligence reguires making a seasoned judg- 
ment, whenever information is combined from different sources and 
circumstances, as to how the various data registered the world, how 
they can be soundiy assessed, and what is reguired in order to inte- 
grate their different perspectives in a way that is accountable to the 
same underlying world. 'Ihe issue applies at all levels, from the most 
far-fung collections of international databases to immediately adja- 
cent posts in the same Twitter feed. Moreover, the integration task 
can never be fully delegated to a meta-level dictionary or translation 
scheme. Ihose are merely more registrations— partial, skewed, and 
contextualiy appropriate in their own ways. If a system is not itself 
capable of jadgment, then it is we who bear responsibility for the dif 
fering perspectives and prejudices of all of its registrational sources, 
and for the legitimacy of all of its instances of data integration. 

Contraty to current fashion, the mere ability to statistically com- 
bine data assembled from diverse sources without exercising judg- 
ment at each and every step may actualiy militate against, rather 
than for, anything we should want to call intelligence. 


for example—if my perceptual system were to de- 
liver that hypothesis to my cortex—I would not 
believe it, would not take the evidence as compel- 
ling. Instead I would concdude that I had blinked 
without realizing it, or that someone had jostled 
my desk, or that what I drank a moment ago was 
not coffee, or that an earthguake was underway, or 
something like that. That is, I would recognize that 
something “impossible” seemed to happen, but be- 
cause impossible things do not happen—and this 
is the crucial fact—I will take that apparent impos- 
sibility as evidence that something has gone wrong, 
that some mistake has been made.” 

In a way, this is no more than Kuhnian normal 
science, We operate within a registration scheme, 
hold objects and phenomena to account as being 
legible in its terms, take suggestions that they are 
not legible as evidence that something has gone 
wrong, dig in to repair the mistake, so as to be able 
to find out something that is correce—and thereby 
increase our knowledge. But in the present context, 
it is also relevant to why Al systems will have to be 
genuinely intelligent if we are to rely on them for 
general conclusions, because they have to be able 
not only to operate within registration schemes 
(ours or theirs), but to hold those registration schemes 
to account throughout their use of them, lest what they 
represent or register parts company with what is or 
could be the case, 


12, Ihis does not mean the world would not ultimately win, if this 
had in fact happened. Rather, the point is that it takes stronger and 
stronger evidence to adjust our sense of the world, proportional to 
the extent of how fundamentally our understanding must be revised. 


Put it this way. Ihere is no “right” ontology—no 
perfect registration scheme. As detailed in the side- 
bar on page 90, this fundamental fact has enormous 
implications for what it is to be intelligent, and for 
how we should view any system not capable of ex- 
ercising judgment. 


6. COMMITMENT: Not only must a system be able to 
distinguish appearance from reality—right from 
wrong —but it must care about the difference. Ihis 
isa point on which Haugeland argued especially 
vehementiy: doing the things we are talking about 
reguires commitment. No creature—neither we nor 
the systems we build—can just incidentally come 
to know what is the case, can just happen to treat 
things as objects. 

For an Al system to register an object as an ob- 
ject, that is, not only must there must be right and 
wrong for it, but that difference must matter, to if, 
Having systems do things that are right and wrong 
for us, having their actions matter to us—that is 
easy. Ihat much is true of calculators, GPS devices, 
databases, and airplane guidance systems. Having 
things be right or wrong for us seems to be enough 
for route planning, and even for landing planes; it 
may be that it will be enough for driverless cats, * 
But it will not give the system itself objects, or a 
world, or intelligence. 

For a system to care, its orientation to the world 
must be backed by a complex constitutive web of 


13. Ie will surely be enough in sufficiently structured and contained 
environments. At issue is whether it will be enough to pilot cars in 
the midst of dense human activity. See the last paragraphs of section 
6 of chapter 11 (pp. 126-127). 


normative commitments. İhe system (knower) 
must be committed to the known, for starters. That is 
part of the deference: to take an object as an object, 
one must defer to the object, in order to sustain the 
appearance-reality distinction on which knowledge 
and intelligence, to say nothing of actuality, depend. 
But the commitments also put conditions on the 
knower. We must be such that we are committed to 
tracking things down, going to bat for what is right. 
Not only are we beholden to the objects, as Hauge- 
land would say; we are also boynd by the objects. 

Explaining the reguisite commitment draws us 
into existentialism. Being in the world, fınding out 
what it is like, ensuring that what one thinks is in 
fact the case, and so on, reguires existential commit- 
ment, without which the whole apparatus would 
dissipate, and ones thoughts or representations lose 
all of their significance— floating free of reality, like 
frictionless pucks in the void. * Commitment to the 
world, that is, constitutes the knower as a knower. 
'Ihe systems we currently use do not need that com- 
mitment, because we have it, and it is we who use 
them. But therein they fail to know, fail to be genu- 
inely intelligent. 


7. SELE; Öne more ingredient is necessary. An under- 
stander— human, Al, whatever—cannot take an 
object to be an object until that understander takes 
itself to be a knower that can take an object to be an 
object. Ihat is: a certain form of “self awareness” is 
necessary in order to achieve the reguisite detach- 
ment to be able to see an object as other, in order 


14. John McDowell, Mind and World, 1996, 11. 


for the system to hold itself accountable for being 
detached and holding the object to account. Again, 
the point is philosophically familiar, but typically 
phrased in daunting language (as Haugeland says: 
“Any disclosing is at once a disclosing of Dasein it- 
self and a disclosing of the being of entities””). But 
we can put it simply. In order for us to know about 
an object: 


a, It must be here in the world, 
b. We must be here in the world, and (recursively) 
c. We must know that both we and it are here. 


It may seem surprising that the full substance of these sev- 
en points is not only something we profoundIy depend on, 
but something that is reguired in order for us to identify 
the most mundane object as an object. But sure enough 
we do, and it is. Surprise merely indicates how deeply the 
ontological assumption underlying fırst-wave Al has been 
engrained in our consciousness—not just inscribed in our 
individual minds, but perhaps incorporated into the world 
view that undergirds contemporary technological society. 


'Ihe failures of first-wave Al, and the fact that second-wave 
Al does not deal with them, should not be taken as under- 
mining the magnitude or importance of the notion of an 
object in our lives—in our finding the world intelligible, in 
our being intelligent, in our ability to navigate, reason, and 
cope. On the contrary, it should heighten our awareness 
of the magnitude of the historical and sociocultural forces 
that have made objects so powerful and ubiguitous. Ihat 


15. John Haugeland, “Truth and Finitude,'in Dasein Disclosed (Cam- 
bridge, MA: Harvard University Press, 2013), 190; emphases in 
original, 


is not to suggest that the notion is innocent. As diverse 
cultures, poets, constructivists, and myriad others know, 
there are situations where objectification and reification 
are gravely problematic.” But for good or bad—most 
likely for both—taking the world to consist (perhaps 
among other things) of objects is a powerful ontological 
or registrational frame. No attempt to build synthetic in- 
telligences can get very far without engaging with what it 
involves in its full substance and gravity. 

With respect to AJ, the success of machine learning has 
taught us that taking the world to consist of objects is not 
a necessary presupposition of synthetic or computational 
devices. More seriously, it has also shown that no genu- 
inely intelligent system can start with an object ontology. 
If a system is to do justice to the world around it, if it is 
to understand what it is representing and talking about, it 
needs to be constructed or evolved in such a way as to earn 
the ability to register the world in terms of objects: objects 
grounded in the world, objects integrated into an unut- 
terabiy rich metaphysical plenum— objects, as we will see, 
held relentlessly to account. 


16. Needless to say, there are problematic particular objects in the 
world, too: nuclear weapons, lies, a few people some of the time. At 
stake here is the more general issue of what justice is done to the 
world to register a patch of it as an object. 


g — World 


Ünderlying all the foregoing points—undergirding the 
conditions on genuine general intelligence—is something 
even more ptimordial. It has to do with that which I am 
calling the world. 

In a way, the point is simple. Everything we believe, ev- 
erything we take in, everything we represent and are com- 
mitted to, must be something we can understand as being 
in a single world—the world that both we and it inhabit. 
Even if a phenomenon makes sense 'on its own” (whatever 
that might mean), it cannot exist, cannot have the regui- 
site actuality and otherness and so forth, unless it is part of 
the world— part of all that there is, that which is total, the 
“One. (See the sidebar on the next page.) 

'Ihis generates a four-fold condition on our taking an 
objece—or anything else—to be real: 


Four-fold Commitment 


1. We must hold the object accountable to being part 
of the world; 

2. Reciprocallyy we must hold the world accountable 
to hosting the object; 

3. We must also hold ourselves, and our relationship 
to the object, accountable to being in that self-same 
world; and 

4. Reciprocally, we must hold the world accountable 
to hosting 4s, and that relationship, as well. 


If something appears to us that is unaccountable, some- 
thing is profoundiy wrong. We have to demur. We cannot 
go there. We must fight like hell to get out of that place, or 
we will die. 

A few explanatory comments. First, honoring this 
standard does not teguire being aware of the four com- 
mitments, especialy in any explicit sense. In the first in- 
stance, in fact, an explicit belief to the effect (especially a 
propositional representation of them) would not help. Ihe 
four-fold commitment is prior; it must grip us as a govetn- 
ing norm, in order for any beliefs or states of awareness 
to be about their subject matters. Second, the “we” in the 
formulation does not refer to us as individuals, even if it 
ultimately places responsibility on individuals' shoulders. 
'Ihese are conditions that societies and cultures have hewn, 
over many centuries, into which we are indoctrinated as 


Pluralism 
Some will say that different people live in different worlds. I am sym- 
pathetic to pluralist ontologies (registration schemes, I would say, 
which confıgure ontologies). But there must be a “lower level” or, in 
some other sense, a more ultimate metaphysical unity holding ev- 
erything together. If X shoots Y “in X's world," Y (that is, that which 
we or X register as Y) will likely die no matter what “world” they 
inhabit. If that which I register as a black hole destroys my world, 
it will likely destroy yours as well. If I want to reach out to you, not 
only would I be unable to reach you, if you genuinely lived in a difter- 
ent world; I would not even know of your existence. And so on. This 
is why reference (like a bullet) must go on through” the registration 
scheme to reach the world itself, See section 12.a. 

As 1 put itin On the Origin of Objects, doing justice to Al, to the 
human condition, and to the world reguires embracing just the right 
combination of ontological pluralism and metaphysical monism. 


children, in learning a language and taking our place as 
members of our social communities. Ihird, the commit- 
ments constitute a high standard. None of us meets them 
all at every instant of our lives, and we certainly need not 
be continuously conscious of them, but as I will presently 
argue, if we are adults we are accountable to them, and the 
civilized and cultural fabrics that enmesh us need to sus- 
tain the standards as parts of a civilized society. 

To put this in cognitive science terms: to take an object 
to be an object does not just mean interacting with it, solv- 
ing the'symbol grounding” problem in a local way, by asso- 
ciating uses of its name with that which it causally interacts 
with—the way an animal might cotton onto prey, or onto a 
rag doll, or the way an infant might attach to their mother. 
Ihough that kind of counterfactual-supporting interactive 
connection may be enough for some purposes, it is not 
enough to constitute something as an object for the sys- 
tem in guestion, and therefore not enough to warrant the 


1. Such as to support a Dretskian information link (see Fred Dretske, 
Knowledge and the Flow of Information, Cambridge, MA: MIT Press 
1981). 

2. An animal will certainly recognize that which we register as an ob- 
jectin some sense—it is the 'objecthood” that is demanding, of which 
I suspect animals are not capable. Strawson (Individyals, London: 
Methuen, 1959) describes a simpler form of registration in terms of 
features—roughly, property-like universals that do not reguire dis- 
crete individuated objects for their exemplification. A standard ex- 
ample is our interpretation of such phrases as “its raining,' which, on 
his account, takes the world to instantiate the feature “raining” with- 
out there needing to be any object that is raining, Ihe suggestion is 
that pets may do something more of this sort— “Its Tiggering again.” 
Ic is even possible that babies initially recognize their parents along 
similar lines: “Hurrah! More mama!” See Ruth Millikan, “A Com- 
mon Structure for Concepts of Individuals, Stuffs, and Real Kinds: 
More Mama, More Milk, and More Mouse” Behavioral and Brain 


systems being called intellişent. We observers might take 
such behavior as legitimating semantic interpretability; we 
might take that with which the system interacts to be an 
object. We might even say that the symbol is 'grounded” — 
that it has determinate semantic interpretation, that being 
that symbol/s interpretation is not a matter of mere whim. 
But throughout, in such cases, the whole pattern remains 
our registration— our object, our deference, 

Ihree examples of failure in the four-fold commitment 
will illustrate. Many years ago, after a parapsychology talk 
at Duke's Rhine Research Center, the speaker asked a few 
of us, on our way out the door, why no one believed his 
results— given, he claimed, that his statistics were as good 
as any published paper in reputable psychology journals. 
“Itisnot that your statistics are bad, Güven Güzeldere ob- 
served, astutely.“Ihey could be ten times better, but still no 
one would believe you. The problem is that no one has any 
idea of how what you are saying could be true” The speaker's 
claims may have been individually coherent, but, as Gü- 
zeldere pointed out, overall they were unaccountable; they 
did not fit into, or consetve, our sense of the world being 
the world. And the world being the world is utterly essen- 
tal, Sans that, all bets are off” 


Sciences 21, no. 1 (1998): 55—65; and“Pushmi-pullyu Representations,” 
Philosophical Perspectives 9 (1995): 185—200. 

3. Ihe point is not to bar hypotheses or observations that challenge 
deeply held worldviews. Some of the greatest scientific advances have 
come from just such cases (black-body radiation, for example). Rath- 
er, what troubled Güzeldere, and myself, was that the speaker was not 
prepared to shoulder responsibility for the fact that what he wanted 
us to believe ran counter to the entire world view on which we all 
relied. What the audience was keenly aware of, to which the speaker 
seemed oblivious, was the enormity of the epistemic and ontological 
burden that would need to be taken on—at the very least acknowl- 


Dreams, to consider a second example, are to my way 
of thinking unaccountable in just this way. In fact I believe 
that it is their very unaccountability that makes it evident, 
from the outside, that they are dreams. lam in aroom with 
two people; it morphs into an auditorium where someone 
is giving a talk; the podium they are leaning on is actually 
coffee cake, which I am eating as I ride my bicycle through 
the Pyrenees. Whatever! It makes no sense, but that does 
not matter, Ido not go to bat to resolve the inconsistencies; 
my heart rate does not soar because my grip on reality has 
loosened. 

A third example. Late one night, in graduate school, I 
was watching a movie in a shared house. Ihe phone rang, 
startling me because it was about 1:00 a.m. | reached over 
and picked up the handset (this was the era of phones 
plugged into walls), whereupon, instantaneously so far 
as I could tell, the television went off, all the lights in the 
house went black, the entire neighborhood was plunged 
into darkness—and the phone kept ringing! In retrospect 
the whole thing was perfectiy explicable;* relevant here is 
merely the intensity of my panic. My heart rate spiked be- 
cause my world seemed to break. Though tiny, it was a ter- 
rifying crack. The threat of losing the world is mottal.” 


edged, and ultimately addressed—in order to make his case credible. 
4. By coincidence there was a city-wide blackout at the instant that | 
picked up the phone. Also by chance, the handset I picked up, left bya 
departing roommate, was not plugged into the wall. 'Ihe original and 
continued ringing came from another phone in the room. 

5. A split second after the incident, after 1 realized that I was still in 
the living room, that the power had gone off, etc., I was overtaken 
by a much more mundane form of fear: of intruders, of the threat of 
violence, etc. But that was subseguent to the prior existential fright, 


I said above that an integrated network of commitments 
undergirds our ability to take an object to be an object— 
to take something to be in the world, to be intelligent. It 
is not just that we are beholden to objects, and bound by 
them. We are also beholden to, and bound by, the world as 
a whole.” 

What does it mean to be committed to the world in this 
sense? | have already said that to take an object to be part 
of reality we have to keep in mind what is true, what is not 
true, and what is impossible. Underlying that is a parallel 
set of norms and commitments about the world as a whole. 
It is not just objects and local phenomena that we have to 
register and fınd intelligible. Registering an object means 
finding it intelligible in terms of the rules and regularities 
that constitute the domain within which it derives its exis- 
tence as the object that it is. But those rules and regularities 
and practices need to be accountable too. Ihey must sup- 
port, rather than undermining, the world's status as world. 

Does that mean that we hang onto our constituting 
conceptual frameworks or registration schemes absolute- 
Iy—because if we let go of them we die? Almost. We lean 
in that direction, We have to—in order to maintain our 
ability to distinguish appearance from reality. But we do 
not hang on to these constituting schemes tyrannically. We 
cannot —and we must not. Just as objects or entities are only 
found intelligible in terms of conceptual or registration 
schemes, so too are conceptual or registration schemes. 
'Ihe constituting rules and regularities underlying practices 


6. That does not mean, at all, that the world is an object. It is not, and 
cannot be. If this were a different era, we might call it GGd— at least 
in Tillichs sense of “God” beinga name for “the ground of being." The 
ground of being is roughly what I mean here by the word “world” — 
that to which I defer, that which is One, that which wins. 


and regimes are only legitimate (that is, as the etymology 
betrays,'can only be read”) if they make sense of the world as 
world. As well as holding objects accountable to constitu- 
tive regularities and norms, so too we must hold constitu- 
tive regularities and norms accountable to the world as the 


ground of being. 


10 — Reckoning and Judgment 


İc is finally possible to explain the title of the book. 

I have tried to sketch some of what is necessary for a 
system, human or machine, to be able to think about or be 
oriented toward the world beyond it—a world with which 
it copes, through which it navigates, in which it conducts its 
projects, about which it reasons, to which it is committed, 
to which it defers. The system must not only be embodied 
and embedded in this world; it must also recognize it 4s 
world. No less is reguired in order for it to distinguish ap- 
pearance from reality, and choose reality; to recognize the 
difference between right and wrong—and choose right; to 
distinguish truth, falsity, and impossibility—and choose 
truth. No less is reguired in order for it to register entities, 
phenomena, people, and situations as such. It must register 
all these things in the world to which it holds itsel£, and all 
that it understands, accountable. 

Such, to our inestimable benefit, is the legacy of the hu- 
man epistemic achievement. 


10a - Animals 
I have not yet said anything about animals. 

Nonhuman animals, I take it, are in one sense account- 
able to the world. There is a huge variety of kinds, needless 
to say; the forms of accountability pertinent to cockroach- 
es will differ substantially from those applicable to jaguars, 


or to chimpanzees. Just how much variation there is—how 


much difference, how much similarity—I leave to others. 
And any such account must include the fact that we are 
animals too. No theory of humans and other animals can 
ignore the continuity of our constitution, evolution, and 
development. 

How much we overlap with nonhuman animals (espe- 
cially higher primates) on capacities that contribute to our 
intelligence is another subject on which 1 profess no spe- 
cial expettise. It is certainly true that nonhuman animals 
manifest species-appropriate forms of awareness. Some 
have exguisitely subtle and precise perceptual (or 'percep- 
tual) systems. There are stakes for them; if they get things 
wrong, they may not survive, individually or as a species. 
Ihey manifest their own forms of care, and are emotional- 
İy sensitive. As is increasingly temarked, in certain respects 
nonhuman animals outstrip us humans. 

Nevertheless, I do not believe that nonhuman animals 
engage in the sorts of existentially committed practices of 
understanding of the world as world that I have been ex- 
ploring here. It seems unlikely to the point of impossibility 
that they are capable of achieving the sorts of objectivity 
necessary in order to take an object as an object, and there- 
by to be truly intelligent in a human sense. We may say 
that a pet recognizes some well-loved object; but that is an 
object for ys. Without the four-fold structure of normative 
commitment aid out above, which a pet seems unlikely to 
be bound by, it cannot be an object for them.” 


ı. Presumabiy backed by massively parallel neural networks, 

2. Or maybe pets and creatures can and do register objects as objects, 
in atleasta rudimentary sense. My point is not to defend any particu- 
lar account of animals, but to get a sense of what is reguired in order 
to take there to be objects, and a world—a sense in terms of which 
we can then have a discussion of whether, and if so to what extent, 
various types of animal and machine can do that, 


Two things can be said for nonhuman animals, though, 
even if they are incapable of registering an object as an ob- 
ject. First, they participate in the world— perhaps even ex- 
istentialİy, at least in a rudimentary sense, each in whatever 
sense that makes sense for them. Second, and crucially, to 
the extent that they represent or register the world, they 
do so in ways that mesh with bow they engage, navigate, cope 
witb—and are vulnerable to—it. The world is presumabiy 
intelligible to them, even if not as world, exactly in as much 
as, and to the extent that, it matches their forms of life, 
stakes, biological needs, etc. 

What matters about animals, that is—and for the pres- 
ent discussion as well—is the (evolutionarily hewn) fıt be- 
tween (i) how they take the world, or whatever patches of 
it they take, and (ii) how they conduct their lives, what they 
care about, and what they are vulnerable to. Ihat makes 
something about their lives or even cognitive powers—if 
we want to use that phrase for nonhuman animals— 
authentic. 1 believe that much posthumanist writing over- 
estimates their epistemic powers, but to the extent that 
animals do something like thinking and understanding, it 
would certainly seem that they do so in an authentic way. 

I will use the term creature for systems, incduding ani- 
mals, whose registrational powers do not extend beyond 
the ways in which they conduct their lives, what they care 
about, what they are vulnerable to, what they are existen- 
tally involved with. 'Ihe term is useful because the devel- 
opment of Al-based “pets” (such as Sony Aibo) suggests 
that we may be, or anyway may soon be, constructing com- 
putational creatures in this sense. As 1 have said a num- 
ber of times already, it seems not impossible that synthetic 
creatures based on machine learning, active sensors and 
effectors, and so on, might not achieve something in the 


realm of the authenticity that I have said animals have 
evolved to possess. That is, nothing in this book argues 
that it may not be possible to construct genuine compu- 
tational creatyres (even if what it would be for constitutive 
norms to apply to synthetic pets, not merely for them to be 
at the mercy of our whims, remains obscure). 


1ob - Computers 
The same is not true for computers in general. Most of the 
computational systems we construce—including the vast 
majority of Al systems, from GOFAI to machine learn- 
ing—represent the world in ways that matter to us, not 
to them. It is because of that fact that we call them com- 
puters, or information processors, and also because of it that 
they have power in our lives, that they matter to us. What 
limits them is that, so far, nothing matters to them. To use 
a phrase of which Haugeland was fond: they dont give a 
damn? Things will only matter to them when they develop 
committed and deferential existential engagement with the 
world (Dasein, perhaps, if we wanted to speak that way). 
A possibly useful diagram of the difference is given in 
figure 8S—a structure in terms of which to understand hu- 
mans, nonhuman animals and other creatures, and most of 
the computers we have built. Ihe forms of life that I have 
labeled “authentic” lie in the region of creatures, animal or 
machine, that register the world (or that we register as reg- 
istering the world) in ways that are appropriate to, and do 
not exceed, the forms of their existential engagement in the 


world. 


3. Zed Adams and Jacob Browning, eds., Giving a Damn: Essays in 
Dialogue with Jobn Haugeland (Cambridge, MA: MIT Press, 2016). 

4, Whatis crucial to the creatures engagement is not the registrations 
themselves, but #be world being as the system registers it, and the crea- 
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Figure 8 


One way to sense the difference is to consider how dif- 
ferent are our interactions with nonhuman animals from 
those with typical Al systems, the outputs of data mining 
services, and so on. Ihough we talk to animals, the fact 
that they do not use high-level language” keeps present 
before our minds the fact that their repertoire of feelings, 
understandings, projects, and the like, remain constrained 
to the realms in which they are able to participate authenti- 
call.“ 'Ihat is exactly not true when our iPhone 'tells' us 
that there has been an accident on the highway, a medical 


tures being existentialiy at stake to that fact. 

I would not begin to suggest that this characterization could stand 
as a definition of authenticity. It is trivial to construct examples of 
systems that meet the criterion of being “under the line” in figure 12 
that no one would count as authentic. 

5. Atleast, so far as we know, they do not use compositional, produc- 
tive, systematic languages (chapter 7, note 1, pp. 71-72). 

6. İf we register a nonhuman animal, such as a pet, in ways that do ex- 
ceed that which is appropriate for how it engages with the world, and 
do so without intending to be metaphorical or ironic— such as when 
I say “my dog is frustrated with postindustrial society too"—then we 
are guilty of attributing a nonauthentic state to it, 


system 'claims' that nausea affects 32.7 of patients who 
take a given drug, or a wearable monitor 'suggests ' that if 
you have another drink you should not drive home. The 
fact that these systems 'use natural language! can obscure 
to us the fact that they understand neither traffic nor nau- 
sea nor drunk driving. What is essential is that the increas- 
ing familiarity of such cases not blind us to the profound 
difference in types of system. 


As noted in the introduction, 1 use the term “reckoning” 
for the representation manipulation and other forms of in- 
tentionally and semantically interpretable behavior carried 
out by systems that are not themselves capable, in the full- 
blooded senses that we have been discussing, of under- 
standing what it is that those representations are about— 
that are not themselves capable of holding the content of 
their representations to account, that do not authenticaliy 
engage with the world's being the way in which their rep- 
resentations represent it as being. Reckoning, that is, isa 
term for the calculative rationality of which present-day 
computers (including second-wave Als) are capable. 

I reserve the term “judgment/' in contrast, for the sort 
of understanding 1 have been talking about—the under- 
standing that is capable of taking objects to be objects, that 
knows the difference between appearance and reality, that 
is existentially committed to its own existence and to the 
integrity of the world as world, that is beholden to objects 
and bound by them, that defers, and all the rest.” 

'Ihe terminology can be justifled in various ways, beyond 


7.It is because they do not exemplify this suite of properties, as 1 said 
in chapter 7 (p. 76), that current computer systems, including those 
constructed on the principles of second-wave Al,'do not know what 


they are talking about. 


Hobbess claim that “reason is nothing but reckoning'—a 
sentiment that many in Al believe, but with which it is 
dear that I disagree. But my aim is simple: merely to con- 
trast reckoning with what we convey in plain English by 
saying that someone handled a situation with “good judg- 
ment. By judgment I mean that which is missing when 
we say that someone İacks judgment, in the sense of not 
fully considering the conseguences and failing to uphold 
the highest principles of justice and humanity and the like. 
Judgment is something like phronesis, that is, involving wis- 
dom, prudence, even virtue.” 

Why does judgment matter? Ihe answer is again onto- 
logical. As I have been at pains to say, registrations (even 
nonconceptual ones) abstract from the worlds detail— 
they approximate, do violence, privilege some things at the 
expense of others. Ihat is not to deny that registrations 


8. Iwo caveats. First, judgment” is normally used in philosophy to 
signify an act, whereas | am using it here to name a property or capac- 
ity—something a system can have. Second, I am not here arguing 
that the notion of judgment being defended in this book is a'mark of 
the mental," in the sense of being a property that all human cognitive 
activity must manifest at all times, in order to count as human cogni- 
tive activity at all, Ie is important that we be able to say that someone 
did or said or thought something without judgment—by reflex, for 
example, or “thoughtlessiy.' 

9. Phronesis traditionally concerns practical judgment, which is not 
my special concern here. Nevertheless, I take it that knowing how to 
act appropriately reguires being grounded in concrete situations and 
recognizing that their subtlety and complexity may outstrip any reg- 
istration of them—a theme | believe to characterize all judgment. In 
general, holding registration accountable reguires resolute commit- 
ment to the world, not to registrations of the world—a theme that 
at least resonates with virtue ethics' deprecation of deontology (see, 
e.g, Alasdair Maclntyre, After Virtue, Notre Dame, IN: Notre Dame 
University Press, 1981). 


are essential. Without them, given our fınite brain (system) 
capacity and the world's partial disconnection, we would 
be at aloss, unable to orient to anything beyond our effec- 
tive grasp. We would have no world.“ Yet no matter their 
necessity, our registrations are not what matters—what 
matters is that which we register. To be accountable, to hold 
things to account, is to know the difference between the 
two—and to be committed to the latter, not the former, 
Think of why we entrust child care to adults” —and 
about what can be catastrophically wrong with the re- 
sponse “But I did everything you said!” We want sitters to 
take care of our children, not of their (or our) registrations 
of our children. We want them to deal with any circum- 
stances that arise, not just with those circumstances we 
have registered in advance. To reach out to the world in 
this way reguires judgment, reguires being an adult.” Or 
similarly: think of how pernicious it is to live with some- 
one who is in love with their image (registration) of you, 
rather than with you “as you are. Or again: if a driverless 
car makes a mistake, what is vulnerable is the person rep- 
resented in their data structures—the full person of ineffa- 
biy surpassing richness, worth, and detail, far transcending 
anything the car could register or conceive. If we are going 
to drive cars that can shoulder commitments, that is what 
we want them to be committed to: the survival of ourselves 
and of others, not the execution of mechanical steps that 
their designers once thought would be the best calculative 


1o. That is, nothing would be world for 4«—nothing would exist, for 
us. “Ihe world itsel£/” as it were, would continue untroubled. 

ıı. At least to emerging adulte—and for that reason. 

12. Ihat is not to say that adults necessary succeed; or that how 
situations are registered is irrelevant to laws and justice. That is all 
secondary. What matters is the child; it is the child who might die. 


route to that end. If they are incapable of such commit- 
ments, then they are mere reckoners, and we should un- 
derstand and deploy them accordingiy. 

If representations and registrations came with God- 
given guarantees of suffıciency, judgment might not be so 
difficult, or so important. But they do not. And they can- 
not. And the world is not such that they could. Ultimately, 
you cannot deal with the world as world without knowing 
that— without judgment. 


Tı — Discussion 


Seven comments. 


I, HUMAN vs. MACHINE: As mentioned at the outset, 


the distinction between reckoning and judgment is 
not intended to get at, or to mark, any present or 
future distinction between people and machines.' 
For one thing, humans are likely also machines, in 
at least some sense, if the word “machine” is to have 
any merit (see sidebar on the next page). More se- 
riously, I see no principled reason why systems ca- 
pable of genuine judgment might not someday be 
synthesized—or anyway may not develop out of 
synthesized origins. It is not my purpose to out- 
law the Golem—to banish synthetic constructions 
from obtaining the highest reaches of humanity.” 
Also, as 1 have said repeatediy, by no means all hu- 
man cognitive activity, especially in the small, meets 
the standards I am proposing for judgment. Perhaps 
the best we can say—though it is a lot to say—is 
that we hold human adults accountable for their ac- 
tions. Even if they do not invoke full scale delibera- 
tive judgment in their every local move, we believe 
that they should act in such a way that they could 
do so if necessary. More generally, my purpose, in 


1. Jadgment is not a mark of the human. Perhaps it is a mark of the 


humane. 


2. See previous note: the highest reaches of humanity, we might say. 


framing judgment as a category, is to establish an 
appropriate standard on cognition such that it be- 


Machines 

Ihe term “machine” has two connotations at least arguably inappli- 
cable to people: being constructed, and being mechanical, Unless one 
is a dualist, however, the pertinence of the fırst to humans is difficult 
to deny unless one embraces, a priori, the belief that having children 
is not a form of'constructing” them. While that may seem obvious, 
it is not an easy thesis to defend. Needless to say, we do not under- 
stand “how children work” simply by participating in their natural 
procreation, but we increasingiy do not understand the computer 
systems we devise, either— especially when their constitutive regu- 
larities (that which makes them the machines that they are) result 
from machine-learning algorithms, the details of whose “machina- 
tions” we do not grasp. If to be a machine reguires being compre- 
hended by engineers at the level of constitutive regularities, that is, 
then ML-based systems (image 'recognizers', "diagnosis' systems, 
"planners”, and the like) would have to be considered to be outside 
the realm of machines. (And understanding the regularities at just 
some level is not enough to separate us, since we largely understand 
how oxygen, hydrogen, carbon, and so on, work— out of which our 
organic selves are built.) 

As for being mechanical, again unless one is a dualist, we are me- 
chanical in some sense—just not constitytively mechanical (unless 
“mechanical” means some special way of being physical that we are 
not, but as above any such construal is likely to exclude computers 
as well). But then neither are computers constitutively mechanical; 
computation, I believe, is a constitutively semantical or intentional 
notion. So again, if people fail to be machines for that reason, so do 
computers. 

Fither we fall within the scope of the machinic, in other words, 
or, at İeast arguabiy, contemporary computers do not. Neither option 
supports a dear human vs. machine distinction relevant to Al, 


> 


comes an empirical guestion whether any given sys- 
tem (human or machine) can or does meet it. 

Another way to look at this was suggested in the 
introduction. One of the challenges raised by devel- 
opments in Al is how they will —and should—af- 
fect not only our sense of ourselves, but the consti- 
tutive standards on what it is to be human (since 
“human”is in part a normative predicate, not merely 
a biological one”). How should Al affect humanity 
itsel£, that is, not just the world that humans live 
in? Without raising jadgment into too much of an 
ideal, we would hardly go awry i£, by turning reck- 
oning tasks over to our synthetic constructions, we 
could “up the ante” on people—raise the standards 
on what it is to be human—by establishing judg- 
ment of the sort described here as a defining bar on 
what constitutes authentic adult thought. 


DETAJLS: İn suggesting a distinction between reck- 
oning and judgment 1 have said little about what 
judgment comes to in detail. Myriad issues arise— 
such as whether the reguisite values and existential 
commitments that judgment involves are or should 


3. Alligators do not need to try to be alligators. Their claim on al- 
ligatorhood is secure, no matter how indolent, peculiar, or bone- 
crushingly violent. Humans are different: we must strive to be human, 
To commit an “inhuman” act is to fail to achieve the minimal ethical 
standards constitutive of membership in the fellowship of the human 
at least, perhaps even the“good." Homo sapiens is one thing; fully hu- 
man, another. (Alligators need not worry about being“inalligator.”) 
'Ihe category “human),' that is, is normative, not merely biological, 
Tc is afhıliated with good and bad, better and worse— with values, mo- 
rality, perhaps even spirit, those vexed interpretive issues, immortal- 
ized by the tree of knowledge, that Descartes clove away from the 
physical ones, back at the beginning of the Scientific Revolution. 


be culturally local, generically human (humane?), 
or of wider compass.* | have suggested—and in 
fact endorse, but have not defended here—the idea 
that accountability to the world can serve as the 
grounds not just of truth but of ethics as well (ac- 
countability to the “One,' to the preregistered, not 
to the postontological reality that most people take 
the word“world” to name). I have similarly left open 
such guestions as whether the possession of judg- 
ment reguires having a history, being enmeshed in 
a society, and so on. And I have said nothing about 
whether we can, or what it would be to, develop an 
amalgam of people, machines, institutions, practic- 
es, values, and the rest such that the assembly might 
be capable of judgment not attributable to the judg- 
ment of any of its members. And so forth. 

Çuestions of this sort, I take it, should occupy 
our attention once we have established a workable 
understanding of the capacities of the systems we 
are building. My hope is that a first-order carto- 
graphic map of the territory occupied by people and 
Als might free us from paralysis in the face of the 
overarching guestion of “whether intellişent ma- 
chines are coming, and will take over?” Once we have 
conceptual eguipment adeguate to the task of tak- 
ing stock of them— of testing their mettle— we can 
take up the raft of detailed guestions we will need 
to answer, even if only inchoately and provisionally 
at first, in order to reach our aim of sensibly deploy- 
ing these systems in ways that are sound, beneficial, 
practicable, and sane. 


4. See Alasdair Maclntyre, Wbose Justice? Which Rationality? (Lon- 
don: Duckworth, 1988). 


3. CONSCIOUSNESS: It will be argued that some of 
the characteristics I have attributed to jadgment 
have more to do with consciousness than with in- 
telligence—and will be claimed, for that reason, 
that it is unfair to pose them as criteria for AGI, 
Ihere is something that is right about this com- 
ment, but much more that is profoundiy wrong. 
Fundamentally, the objection gets the situation ex- 
actly backwards. 

What is right is that some of the properties of 
ful-blooded intelligence that 1 am describing— 
perhaps especially its dealing with a world of sur- 
passing richness, exceeding the grasp of discrete 
concepts —are typically associated with “gualia” and 
other phenomenal characteristics recognized as be- 
ing beyond the compass of discrete propositions 
and traditional logic. But to think, as a conseguence, 
that the notion of judgment being defended here 
indudes more within its scope than “mere rational- 
ity, and thus must be treading into the territory of 
consciousness (here) or emotion (next section), is 
to miss the fundamental ontological point that has 
underwritten the entire analysis. 

If present-day theories of logic and of rationality, 
based on'clear and distinct” concepts, considered as 
given and unproblematic, do not provide an analysis 
of what it is to take responsibility for those concepts 
(responsibility for how they abstract, and idealize, 
and do both justice and injustice to, the worlds to 
which they apply), then so much the worse for those 
theories of rationality. Rich is how the world is, to 
which reason (not just feeling and consciousness) 
must be accountable. Various are the registration 


schemes suitable for characterizing it, including 
schemes beyond the reach of discrete concepts. 
That the world outstrips these schemes' purview is 
a blunt metaphysical fact about the world— critical 
to any conceptions of reason and rationality worth 
their salt. Even if phenomenological philosophy has 
been more acutely aware of this richness than has 
the analytic tradition, the richness itself is a funda- 
mental characteristic of the underlying unity of the 
metaphysics, not a uniguely phenomenological or 
subjective fact. 

More generally, I have tried to argue that mak- 
ing sense of first- and second-wave Al, on the one 
hand, and of the difference between reckoning and 
judgment, on the other—two aims of this book— 
reguires rejecting a widely accepted divide between: 
(i) a cut-and-dried realm of conceptually discrete 
calculative rationality, of the sort that GOFAI tried 
to emulate, and that has perhaps unduly influenced 
Western metaphysics and epistemology for too 
long, based on a Hawed ontological presupposition 
of a world of unproblematic discrete objects and 
properties; and (ii) a gualitatively richet, more Huid 
realm of “subjective” consciousness attuned to the 
subtleties and nuances of the world's resplendent 
underlying richness.” Second-wave reckoning (not 
even judgment) is succeeding by recognizing the 
world itself as ineffably dense. Judgment reguires 
holding rationality relentlessly to account for this 


5. It is“underlying” only with reference to the short-sightedness of an 
excessive allegiance to conceptual abstraction. As machine learning 
should if nothing else alert us to, there is nothing “hidden” about the 
worlds indelible variety. 


abundance, constantly ready to relinguish alle- 
giance to registration schemes when they do it too 
much injustice. None of these things have uniguely 
to do with the first-person subjective character of 
cOnsciOUsness, 

Put it this way: logic and GOFAIJ failed by 
viewing rationality as a postregistration—'post- 
ontological” — process of articulated moves, on the 
model of a chess game. Machine learning is helping 
to open our eyes to what phenomenologists have 
long understood: that holding registrations ac- 
countable to the actual (nonregistered) world is part 
and parcel of intelligence, rationality, and thought. 
If these insights lead to a reworked understanding 
of rationality, teason, and thought, which in turn 
impinges on and adjusts our understanding of con- 
sciousness (as 1 believe it should and will), well and 
good. But that is the direction of the implication: 
from the nature of reality to consciousness, not the 
other way around. 


> 


EMOTION: İhe distinction between reckoning and 
judgment is also not a difference between reason 
and emotion. Some may feel that being commit- 
ted, going to bat, giving a damn, and so on, must be 
emotional states, in virtue of being action-inducing 
and motivational. For many reasons, 1 believe this 


6. Consciousness does have its peculiarities, including its ineliminabiy 
first-person, unshareable character, which I will address elsewhere 
(e.g., in“Whos on Third? The Physical Bases of Consciousness,' un- 
published manuscript). I believe, however, that the identification of 
phenomenal gualia as uniguely characteristic of consciousness and as 
scientificaliy inexplicable are both mils, deriving from an ontologically 
inadeguate understanding of the nature of reality. 


is radicaliy incorrect. Authentic judgment reguires 
detachment and dispassion, for starters” —freedom 
from some of the vety vicissitudes most character- 
istic of emotional states (“set your emotions aside,' 
we appropriately tell children, when guiding them 
toward considered judgment). Interpersonal com- 
mitments, too, must similarly reach well beyond 
emotion, in my book, if they are to endure. 

But the present point is not to impugn emotion, 
which some will argue to be essential to achiev- 
ing humanity, instilling compassion, and so forth. 
Rather, as suggested above, I want to reject the 
idea that intelligence and rationality are adeguate- 
İy modeled by something like formal logic, of the 
sort at which computers currentiy excel. That is: I 
reject any standard divide between “reason” as hav- 
ing no commitment, dedication, and robust engage- 
ment with the world, on the one hand, and emo- 
tion and affect as being the only locus of such “pro” 
action-oriented attitudes, on the other.” It would be 
ruinous, | believe, to phrase this in terms 1 owe to 
Charis Thompson,? to cede logos to reckoning, and 


7. See the introduction, note 2 (p. xv). 


8. Among other issues, if one draws into the same category of emotion 


not only dispassionate concern with the truth of ones thoughts, the 
adeguacy of one's registration, and the detachment of ones perspec- 


tive, but also self-indulgence, bias, prejudice, self-interest, revenge, 
and so on, one would then reguire the wherewithal with which to 


distinguish “good” emotions, as it were, from “bad” (the former from 


the latter)—a discriminating task, in my book, that would need to 
rely on dispassionate judgment. 


9. Charis Thompson, personal communication, 2018. See the intro- 
duction to her Getting Ahead: Minds, Bodies, and Emotion in an Age 
of Aytomation and Selection (forthcoming). 


force judgment to take refuge in pathos. Even the in- 
creasingiy popular suggestion that “good thinking” 
reguires combining rational understanding with 
emotion and affect buys into this impoverished 
conception of rationality. 

Asan example of such a misconception, consider 
Ihomas Friedman's suggestion “ that since indus- 
trial era machines replaced work of human hands 
(physical labor), and second-wave Al systems are 
now taking over the work of human beads (mental 
labor), what remains for humanity is labor of the 
heart: passion, character and collaborative spirit. 
Ihough superficially elegant, the suggestion is fatal- 
İy Hawed. It 'disappears” the deliberative judgment 
that I am arguing is fundamental to reason itself, 
and is thereby blind to anything approaching genu- 
ine intelligence or dispassionate inguiry. Determin- 
ing, standing for, and going to bat for the truth—all 
foundational components of reason—reguire ac- 
tive and 'resolute commitment,' as Haugeland puts 
it, far beyond anything computers can yet do. 


5. RESPONSIBILITY: Fifth, only a system capable of 
authentic judgment can truly shoulder responsibil- 
ity for its actions and deliberations. If an automated 
guidance system causes an airplane to miss the run- 
way and to slam into a mountain, it is unexceptional 
to say that the system failed; we might even say that 
it was at fault (though I would want to write as 'at 
fault). But we would—and should—balk at hold- 
ing any presentİy constructable computer system 
responsible for the tragedy. An ethical stance can be 


10."From Hands to Heads to Hearts, New York Times, Jan. 4, 2017. 


shouldered only by an existentially committed sys- 
tem capable of jadgment. 

Historically, it has been said that responsibility 
can be shouldered only by a system capable of hbu- 
man judgment. One of my aims in eschewing the 
human /machine distinction is to make room for the 
development of a notion of jadgment substantial 
enough for the gualifer “human” to be neither neces- 
sary nor suffıcient. We need to be able to ask where 
responsibility for a given task is and should be locat- 
ed, without prejudice as to whether that judement 
resides in person or machine (or in policy, practice, 
law, community, etc., or an amalgam of them all).” 
6. ETHICS:; For two reasons, | have intentionally not 

taken up the topic of ethics. It has long been my 
view, fırst, that substantial discussion of ethics and 
Al reguires a deeper understanding than we have 
had, to date, of the issues I have been addressing 
here: what Al is, what intelligence teguires, and so 
on. This book can thus be viewed as an attempt to 
lay out some of the understandings that I take to 
be prereguisite to meaningful conversations about 
the ethics of AT, Second, as this book again demon- 
strates, | do not believe, although it seems tempting 
to some, that approaching the subject of Al through 
an ethical lens is the best route into the deepest and 
most serious issues about its nature, capacities, role, 
impact, and proper deployment. 


ıı. İhis injunction must be treated with care. It would be disastrous 
to water down what responsibility is, what it takes, and the standards 
to which we should hold both people and machines accountable, if 
they are granted it, under pressures to delegate judgment to fırst- and 
second-wave Al, See pages xix—xx of the introduction, 


One preliminary ethical comment can be made, 
though, preparatory to any substantial discussion. 
On the one hand, many of the (critical) ethical 
guestions raised by the development of reckoning 
technologies are unprecedented in detail, but of a 
kind familiar from the deployment of any new tech- 
nology: genetically modified organisms, social me- 
dia, nuclear weapons, and so on. Entirely different 
considerations arise, unprecedented except perhaps 
in the context of child rearing, when we ask what 
it would be for Al systems themselves to be moral 
agents—that is, to be able (and hence mandated) 
to take ethical responsibility for their own actions. 
I believe we are a long way from this, but the gues- 
ton will arise if and as our creations start to “think 
for themselves” My view is straightforward: such 
systems must be capable of moral judgment, under- 
stOOd as a species of exactly the sort of judgment 
under discussion here. 

If does not follow that an Al capable of being 
a moral agent must for that reason be capable of 
employing ethical concepts explicitly, in the sense 
of holding or reasoning with an explicitly articu- 
lated moral theory. Just as a young adult might 
demonstrate estimable ethical jadgment and be- 
havior (perhaps even preternatural wisdom) in the 
absence of an explicitly held moral theory, so too 
might a synthetic system. Contrapositively, it is easy 
to imagine a reckoning system devised to calculate 
and manipulate symbolic representations of ethi- 
cal worths and values, which did not even have the 
ability to refer authentically to the entities we would 
take its intentional states to represent—a system, 


that is, inherently incapable of ethical behavior at 
all, What grounds ethics, in person or machine, is 
the moral character of the subjects (semi-autono- 
mous) judgment, intentional orientation to, defer- 
ence, and engagement with the world—not, in the 
first instance, its possession or lack thereof of an 
articulated (ethical) theory. 


A postscript might be added on a topic of intense 
current debate: the issue of driverless cars. To clear 
a potential distraction aside fırst, one of the diffı- 
culties of raising'trolley problems” in this context, ” 
perhaps reflected in the fact that humans rarely 
know what they would do in such situations, is the 
presupposition that driving reguires deliberative ra- 
tiocination. İhe human case is complex; we build 
roads and constrain traffıc patterns and develop 
complex social practices so as to minimize the cog- 
nitive demands on human drivers; it would actualiy 
be dangerous for human drivers to need to assess, 
anew at every instant, whether the road contin- 
ues around the corner or drops off a diff, whether 
the vehicle in front will be grabbed by a hook ris- 
ing from the road's surface and instaneously come 
to a halt, etc. Human drivers are held accountable 
only within highly regimented domains in which 


12. Ihe original trolley problem, popularized by Philippa Foot but 
originating at the beginning of the twentieth century, concerned the 
difference between doing nothing or intervening to afect the out- 
come of a situation with varying moral conseguences. In discussions 
of driverless cars, the problem is usually simplifed to one of merely 
choosing between two courses of action with distinct moral conse- 
guences (such as killing more elderiy people or fewer children, or giv- 
ing higher priority to passengers than to pedestrians). 


* 


as much judgment as possible has been shouldered 
and prefigured by material arrangements, social 
practices, and norms, 

Ihat said, the issue we face as a society is one of 
confıguring traffıc in such a way as to ensure that 
vehicles piloted by (perhaps exceptionally acute) 
reckoning systems can be maximally safe, overali. 
It may be that in some situations—long distance 
highways, for example— we can restrict the driving 
context suffıcientİy, as we cutrently do for aircraft, so 
that reliable reckoning confers adeguate safety. Ihat 
is, one possibly sane way to deploy currentiy imag- 
ined driverless cars would be to ensure that the vari- 
ous kinds of information available to them can be 
combined, via reckoning alone, to lead to outcomes 
we humans judge to be safe, Then, if and as we are 
able to develop systems that approach anything like 
judgment, the contexts in which they could be safely 
deployed will proportionally increase. But I do not 
believe that automated judgment, as I have empha- 
sized throughout, is on any immediate horizon. 

Ihe conclusion generalizes. We should not del- 
egate to reckoning systems, nor trust them with, 
tasks that reguire full Hedged judgement—should 
not inadvertently use or rely on systems that, on the 
one hand, would need to have jadgment in order to 
function properly or reliably, but that, on the other 
hand, utterly lack any such intellectual capacity. 


CONSTRUCTTON: Finally, I have said nothing about 
how judgment might be constructed. While in de- 
tail I do not believe any of us know, a number of 
points can already be made. 


Note, for starters, three facts already stated 
about judgments constitution. First, all intentional 
and semantic properties, of which judement is an 
instance (along with being right, telling the truth, 
being reliable, etc.) are noneffective, relational prop- 
etties, ” Ihey are not facts about the mechanical 
(effective) shape of what is in the machine.* So 
any guestion about the presence of judgment is a 
guestion about the system “under interpretation,' 
not about its mechanical form, Second, as has also 
been emphasized, judgment is a holistic property of 
how a system works—how things are tied to other 
things, what the system does when encountering 
conflicting evidence, what other events or processes 
one perceptual event will trigger, how it responds 
to situations overall, etc. Even under interpretation, 
therefore, judgment is unlikely to be an isolable or 
categorical property of an architecture. Third, judg- 
ment involves a systems giving priority to the world, 
over its internal states and representations—and 
over the patterns exemplified by any inputs, machi- 
nations, and “interior” processes. What 'giving pri- 
ority to the world” looks like, from an internal archi- 
tectural point of view, is at this point anyone's guess. 


13. Calling them relational properties is technically suspect, because 
in general a relational property would be taken to be one that related 
two (or more) other ontologically secure entities. On the metaphysi- 
cal and ontological world view being suggested here, however, reg- 
istrational properties are partially (not wholly!) constitutive of any- 
thing's being the thing that it is, and so the entities “related” by the 
intentional or semantic properties are not necessarily ontologicaliy 
individuable except in the relational properties terms. 

14. See my forthcoming Computational Reflections for an account of 
how even internal reference, where the target of the intentional act is 
an interior process or structure, is not an efective relationship. 


Several conseguences follow directly. First, the 
presence of jadgment within a system is unlikely to 
be detectable at any level of implementation below 
the full system or personal level. It is unlikely ever 
to be realized by a structure or pattern of activity 
identifable as such within the systems internal ar- 
chitectural configuration. Even less is it likely to 
have an effectively detectable neural or mechanical 
correlate (just as the presence or shape of trye sen- 
tences within a logical system cannot be determined 
purely on the basis of syntactic shape). No #MRI of 
a person, that is, and no analogous description or 
diagram of the effective state of a synthetic system, 
is ever likely to reveal whether the system has any- 
thing approaching reliable judgment.” 

Second, because of the constitutive importance 
of giving priority to the world over any internal 
representations or states, judgment is unlikely to 
be achievable by any system that does not engage 
with the world itself, in ways appropriate to what 
is reguired to hold its judgments to account. Judg- 
ment is available to systems and organisms that par- 
ticipate in the world; it will never be accessible to a 
mere bystander or disconnected reckoner. 

Ihird, as already remarked, I do not believe that 
genuine intelligence can be explained within the 
bounds of science as currentİy conceived— from 
within a methodology of blanket mechanism, that 


15. Someone might argue that there may be neural or mechanical cor- 
relates of jadgment rather along the lines of the presence of a chemical 
or hormone corresponding to existential anxiety, which would per- 
haps not exist in a purely reckoning system—but these are specula- 
tions of fantasy rather than anything rationally defensible at present. 


is— because of (at least current) science's commit- 
ment to physical mechanism and causal explana- 
tion, which blocks its ability to encompass both the 
noneffective reach of semantics and the normative 
nature of existential commitment.” 


It is considerations of this sort that underwrite the 
first sentence in the book—that genuine intelli- 
gence, or anything wotthy of the AGI label, will not 
be achieved with “more of the same”: increased pro- 
cessing power, accelerated practical development, 
new technoscientific research of the sort that has 
been conducted to date. Contemporary research 
tends to focus on effective mechanisms, algorithms, 
architectural configuration, etc.—none of which are 
of the right conceptual sort to lead to insight about 
deliberation and judgment. 

What could lead a system to judgment? Child- 
rearing is an instructive example. Developing judg- 
ment in people reguires steady reflection, guidance, 
and deliberation over many years—interventions, 
explanations, and instruction in situations where 
judgment is reguired, even if that need is not su- 
perfıcially evident. What are those situations, where 


16, Some will argue that psychological theories of memory are legiti- 
mately scientific and crucialiy trafhıc in the situations or phenomena 
that the memories are about (such as childhood experiences). The 
difficulty is that, although such accounts may help themselves to the 
contents of the memories, they cannot, within a mechanistic concep- 
tion of science, explain or give an account of what makes those distal 
in-the-world situations be the memories! contents. (It would not do, 
of course, to say that those are the situations that the subjects de- 
scribe when the memories are invoked or reported on; the problem 
just recurs—how can a mechanistic scientific account explain what 
their statements are about?) 


guidance and instruction is warranted? How is it 
determined that (and what kinds of) intervention 
and guidance are reguired? Ihese are guestions 
with which every parent wrestles on an ongoing 
basis. Ihe issues, symptoms, and recommended 
courses of action are deeply embedded in human 
culture; the patterns of behavior and thought that 
we hew to have been crafted over centuries of civi- 
lization (including in the religious traditions that 
have served as stewards for ultimate guestions in 
all major historical civilizations).Ihe stewards of a 
child's education— parents, schools, religious insti- 
tutions, mentors, İiteratures, communities, and so 
on—bring to bear attention, wisdom, and reflec- 
ton that build on this cultural legacy, skills distilled 
over many generations, and handed down from one 
generation to the next. 

Ihere are reguirements on the child, as well. At 
each stage, a child must have some ability, increas- 
ing as they mature, to reflect on their own behavior, 
to “step outside” the situations they are engaged in 
enough to be able to assess the appropriateness and 
conseguences of acting or responding in various 
ways, and so on. Notabiy, there is nothing in princi- 
ple limited about what sorts of consideration might 
be applicable or worth considering in any particular 
case. İhere are maxims that are applicable; but any 
attempt to put them into articulated form would 
almost surely be vapid and contextually insensitive. 


What does all this mean in the case of Als and 
computer systems generaliy? Perhaps at least this: 
that it is hard to see how synthetic systems could 
be trained in the ways of judgment except by be- 
ing gradualiy, incrementally, and systematicaliy 
enmeshed in normative practices that engage with 
the world and that involve thick engagement with 
teachers (“elders”), who can steadily develop and 
inculcate not just “moral sensibility” but also intel- 
lectual appreciation of intentional commitment to 
the world. 

Is this a realistic route forward for AI? That is 
not for me to say. As | have made clear since the 
beginning, it is not my aim to bring the imagination 
of Al designers to a halt. But if synthetic judgment 
isan end toward which we want to aim, ” these ate 
not only the sorts of goals that will need to be em- 
braced, but also the sorts of methods that may need 
to be undertaken. If that proves to be correct, at- 
taining synthetic judgment will reguire a profound 
shift in the nature of research and a radical wid- 
ening of approach. One of the aims of laying out 
the seven standards in chapter 8 is to suggest what 
would need to be addressed in order to make good 
on that original Al dream. 


In passing, I should say that I expect advocates of 
evolutionary explanation and deep reinforcement 
learning to suggest that we could approach this pro- 
cess with appropriate “reward structures” and/or 
“threats,” designed to mimic cultural and personal 


17. Whether we do or not is a guestion on which I am not taking a 
stand here—though it is hard to imagine current efforts to develop 
AGI drawing to a close, 


development. For at least two reasons, I doubt that 
such strategies will succeed. For one thing, they 
seem liable to drift into calculative reckoning about 
morality, rather than engagement with the world of 
the sort that will lead a system out of commitment 
to its internal representations and toward deferen- 
tial commitment to the world. In addition, it seems 
unlikely that any simple exploitation, by a regimen 
of rewards, of even genuine existential vulnerability 
would suffıce. Famously, you cannot “buy” generos- 
ity or kindness in people through any simple be- 
haviorist pattern of stimulus and response, reward 
and punishment. Ihere is no reason to suppose that 
the situation would be any different for a synthetic 
creature, 

Judgment reguires a fınely-honed sensitivity to 
epistemic and ontological subtleties in ontological 
webs of practices and entities whose importance 
has been forged through cultural and historical 
processes beyond any one petson's comprehension. 
I find it unlikely that judgment could be instilled 
or inculcated in a synthetic device through anything 
less than a moral and ethical upbringing and steady 
process of education, apprenticeship, enculturation, 


guidance—perhaps even friendship. 


12 — Application 


The impact of the seven points laid out in the last chapter 
can be appreciated by applying them to three issues, almost 
technical in nature, of critical importance. 


12a - Reference 

'Ihe fırst has to do with the nature of reference, I have said 
throughout that jadgment reguires holding registrations 
to account. İhis reguires commitment to the world being 
registered. Commitment to the registration is insufficient, 
as | have emphasized throughout; so too, we can now see, 
is commitment to the world as registered. Rather, what is 
reguired is commitment to that which is registered. It is not 
just the difference between registration and registered that 
matters, that is— effectively a difference between sign and 
signifled, between map and tetritory. Rather, the point is 
metaphysically more fundamental: there will always be a 
surplus to that which is registered, beyond its being as it is 
registered as being. You may be a person, but there is more 
to you than how my tegistration of you as a person takes 
you to be. Andit is to that complete you that 1 refer, if 1 
refer to you. lo register an argument as brilliant, I must 
refer to it—whether or not it is brilliant, needless to say, 
but also whether or not it is even an argument. In general, 


1.Ifitis not brilliant, then I will have said or thought something false; 
but for that to be the case what I said must still be a statement, a 
thought, with the reguisite stature to be false, 
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in order to fix on the world, reference and thought must 
“go on through” any registrational schemes used to parse or 
make sense of it, to reach the world itself. It is that patch 
of the world, not the world- 
as-registered, to which our 


commitment binds us, to 
which deference is owed. 
'Ihis is true—and this is the 
critical part, and the difh- 
cult one—even if the crite- 
ria used to individuate what 
we refer to have their roots 
in the registration practices 
employed.” 

As a possibiy helpful 
analogy, consider the Mon- 
i : drian painting shown in 

Figure 9 figure 9, and the descriptive 

term “the large grey rect- 
angle toward the upper left”* Call this term a. Suppose, 
counterfactualy but plausibiy, that figure 10 is an accurate 
representation, at a very high level of magnification, of a 


2. A realist who presumes that object ontology inheres in the world 
independent of human practice can view reference as attaching to a 
pre-existing object, and truth as having to do with whether or not 
that object exempliftes a property daimed of it. That option is un- 
available to anyone with constructivist ontological sympathies. And 
the conseguences of its not being available are profound. It is a chal- 
lenge of the utmost importance to anyone attempting to make good 
on a metaphysical view with constructivist leanings to ensure that the 
commitment of judgment is to that-which-is-registered, not to the 
world-as-registered. 

3. Piet Mondrian, Composition with Red, Yellow, Biye, and Black, 
(1921). 

4. Redin the original. 


12 — Application 


portion of the top edge of that rectangle. At least arguabiy, 
greyish patch A falls within the referential scope of a, and 
dark patch B perhaps not (whether whitish patch C does 
or does not is less clear), even though A and B lie without 
and within, respectively, the boundary of the most plau- 
sible actual (idealized, Platonic) rectangle associable with 
a, Were Mondrian to instruct an assistant to “erase the 
rectangle, for example, it is unarguable that A should be 
removed. 

'Ihe image is just an analogy. Ihe point is that anything 
that is registered (a person, a gambit, a nation) will outstrip 
in richness not only what is grasped about it by its registrar, 
but also what is included within any idealizations used to 
identify it as the referent, and ultimately what is humanly 
graspable at all. Only if that is trye can registrations be held 
to account. The point may sound mystical, but is in fact 
commonsense: that which 
we understand will always 


MM sirict rectangle boundary 


transcend our understand- 
ing of it—including its “it 
ness;” its boundedness and 
identity as an object. 

I said earlier that one of 
the things that differenti- 


ates a reckoning system 


Figure 10 from a system with judg- 
ment is that the former is 
content to operate on registrations of the world, whereas 
the latter must be committed to, must be concerned with, 
must care about the world thereby registered. What this 
last discussion emphasizes is that, in order to be account- 
ably committed to that world, a registering system must 


“tread lightly”—holding its registrations and registration 
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schemes in perpetual epistemic and normative abeyance, 
always prepared to let go of them in order to do justice to 
that which they are used to register. In this case a Buddhist 
phrasing is apt: you cannot reach the world until you “re- 
linguish attachment” to the registrational machinery you 


employ to fınd it intelligible, 


ı2b - Context 

Ihe second point has to do with an issue that has plagued 
Al research since the beginning—in both its first- and sec- 
ond-wave iterations—and that DARPA has identifed as 
criterial for AGI, and imagines being part of “third-wave” 
Al: the notion of context.” Ihe issue is not merely one 
of having a computer system use context-sensitive struc- 
tures and symbols in appropriate ways, such as indexicals 
and perspectival descriptions (analogs of “today,” or “the 
medium in this drive," etc.), but of confıguring a systems 
'deliberation' to be appropriate to the wider situation at 
hand beyond what is immediately represented, either ex- 
plicitly or implicitly—neither ignoring facts or phenomena 
that are in fact pertinent, nor wasting time and resources 
exploring any (or at least very many) of the infınite number 
of alternatives that'“in theory” might be relevant, but in fact 
are irrelevant—and that anyone with a modicum of com- 
mon sense would recognize as irrelevant. 

We can approach the topic by considering the technical 
implications of part of the epistemological critigue of fırst- 
wave Al: that thinking emerges from'a horizon of ineffable 
knowledge and sense-making” To be attuned to context is 
not simply being able to consider other factors from within 
a given registration scheme, or even being able to shift from 


5. John Launchbury, “A DARPA Perspective on Artificial Intelli- 
gence, https://www.darpa.mil/attachments/ AlFull.pd£ 


one registration scheme to another, as if stepping from one 
weli-defined island to another, as in GOFATS image of 
logical reasoning. Dealing skilifully with contexe—and this 
is really the point—is not a postregistrational, postonto- 
logical skill, On the metaphor of chapter 3's figure 6 (p. 34), 
contextual awareness reguires the ability to move continu- 
ously and ineffably in the submarine topology— steward- 
ing the system's embodied and embedded participation in 
the world, registering only if, when, and as appropriate. 

It has long been recognized as an unassailable fact about 
genuine intelligence that anything can be brought to bear if 
pertinent. Ihere is no bound or specifable limit to what 
might be crucial in any given circumstance, relevant in any 
particular line of reasoning. Ihe present analysis ups the 
ante: there is no assurance that any registrational scheme 
will be adeguate to an unexpected circumstance. It is con- 
ceptually impossible, in this English language, to describe 
a situation that cannot be registered, needless to say. As 
a gesture in that direction, however, we can imagine situ- 
ations that put pressure on the idea of any preconceived 
registrational repertoire being adeguate for training ma- 
chine learning systems: a damaged aircraft crash-landing 
over ones head while driving, the ethics of murder in a so- 
ciety where people regularly back up their brains online, 
the nature of a universe with two spatial and two temporal 
dimensions rather than three and one, and so on. 

'Ihe only “ultimate context” is the world as a whole— 
something that no registration scheme will ever encom- 
pass. Sensitivity to context, in other words, reguires far 
more than choosing from a prefigured set of registration 
schemes. Context-sensitivity is not merely a guestion of hav- 
ing a world model, in other words. No such model can be 
adeguate to all potential circumstances. Rather, sensitivity 


to context reguires being able to choose or develop a model 
(registration scheme) adeguate to whatever circumstances 
are at hand, backed by resolute commitment to hold that 
model (scheme) accountable to the world it is a model of. 

Ihe world reguires commitment to keep in view. Con- 
textual awareness must be based on such commitment. As 
such, conceptyal sensitivity reguires udgment. It will never be 
achieved by mere reckoning. 


12c -“ TW 
Finally, it is instructive to take stock of where this long 
analysis has taken us. 

I daimed in chapter 3 that the ontological critigue of 
GOFAI cut the deepest.9 It was not a critigue of ontology, 
needless to say, but of Al systems, having to do with the 
ontological assumptions they made about the worlds they 
were tasked with understanding. In framing it, I used the 
term 'ontological” to mean roughly “what there is"— that is, 
the furniture of the world, what objects, properties, and 
relations exist. 

IT also said, in the introduction, that one of the most im- 
portant contributions of second-wave Al is the fact that 
it gives us a window onto a different ontological view. But 
from where we have reached we can see that that was an in- 


6. Ihe phrase'ontological critigue” is from Dreyfuss What Computers 
Can't Do (1972); he used the term for a critigue of Al similar to mine, 
although, somewhat surprisingiy, he framed his version rather episte- 
mologically —as having to do with information, data about the world, 
understanding, etc., rather than with the structure of the world per se 
(“the data ... must be discrete, explicit, and determinate” (118 |, “every- 
thing essential to intelligent behavior must in principle be ynderstand- 
able in terms of a set of determinate independent elements” (118), “the 
world can be analyzed as a set of facts—items of information” (130), 


etc.; emphases added). 


adeguate characterization. Ihe impact of ML is far greater, 
Ihese new systems, and the experiences we are gleaning 
through building them, are teaching us about ontology it- 
sef—about what objects are, about how individuation 
arises, about what the world is like “underneath” the ob- 
jects (i.e., in more detail than is grasped in the abstractions 
in terms of which the objects, properties, and relations are 
individuated). Second-wave Al, in other words, is provid- 
ing us with insight into the very metaphysical foundations 
of ontology.” I said at the outset that the book would have 
a strongly ontological Havor. Had we had the reguisite dis- 
tinctions in front of us at that point, I could have said: a 
strongly ontological and metaphysical Havor. 

Second-wave Al, we have seen, is demonstrating some- 
thing I have said for many years: that ontology emerges 
in the context of registration practices; it is not the prior, 
pregiven structure of the world.* Moreover, and directiy 


7. NonstandardIy, I use the terms “metaphysics” and “ontology” dis- 
tinctly: “ontology” to refer to what there is—to the furniture of the 
world, to reality as we register it; and “metaphysics” to refer to (and, 
when appropriate, to the study of) that which undergirds ontology— 
the world, the foundations and fundaments of objects and ontology. 
8. Ihis is not to say that ontological items or entities— objects, prop- 
erties, states of affairs, etc. —are ways of taking the world. Tables, de- 
tente, machinists, and love affairs are absolutely not merely epistemic 
entities—not thoughts, not representations, not ideas, not ingredi- 
ents in our registration schemes, not patterns of perception, not infer- 
ential processes, not types of data structure, not any other purely cog- 
nitive or epistemic or machinic configuration. In that sense the view I 
am defending is stubbornly realist: all these entities are things beyond 
us, things in the world. The point, though, is that che individuations 
and abstractions and schemes in terms of which this furniture of the 
world is framed (individuated, categorized, discretely conceptualized, 
ontologically“parsed”) are the result of social and individual processes 
of registration. Tables, ddtente, people, etc., are not“ways of taking the 
world. Ihey are the world taken a certain way. 


applicable to Al, registration, including determining an 
appropriate registration scheme, is among the most im- 
portant characteristics—if not the most important char- 
acteristic— of intelligence itself, To put it bluntly: it is not 
suffıcient for Al, and the broader study of intelligence of 
which it isa part, merely to make assumptions about ontol- 
ogy. It must explain ontology. 

İnvestigating registration, and thus moving the subject 
matter of ontology into the research agenda of artificial 
intelligence, are among the most important philosophical 
and scientific conseguences of recent developments. 


"This much we have learned from ML and second-wave Al: 


1, We need to revise our understanding of concepts, 
and of conceptual reasoning, to appreciate the pow- 
er that concepts derive from being drenched in non- 
conceptual (submarine) detail. 

2. We need to understand registration, and recognize 
registering the world appropriately as possibly the 
most important characteristic of intelligence. 

3. It is insuffıcient for Al, and the broader study of 
intelligence of which it is a part, to assume that 
intelligence is a capacity of systems deployed in 
an ontologically structured world. Ontology is an 
achievement of intelligence, not a presupposition. 


But there is more. What ups the ante for Al is that genuine 
intelligence—intelligence at the highest level, anything of 
the sort that could underwrite judgment—reguires under- 
standing these ontological points, in at least a tacit sense— 
enough for them to both normatively and pragmatically 
govem its behavior. As I have emphasized throughout, 
judgment reguires commitment to what we might call a 


third level of representation: not to the first level of the 
representation or registration itself (not, as I have said 
since the beginning, to a data structure, or to an image of 
a person, or to a description or term); and not even to the 
second level: the world as registered (as a table, machinist, 
etc.).” Lam not convinced that either first- or second-wave 
Al systems have yet made the transition from the first of 
these to the second. But anything approaching judgment 
reguires a third, much more demanding version: commit- 
ment to tbat which is registered a certain way (as a table, 
detente, or whatever). As always, “world” in this statement 
does not refer to what we ordinarily think of the world as 
being—a planet circling the sun, a profusion of social and 
political arrangements, a biological ferment. Rather, the 
world, in the only sense that makes sense for this third lev- 
el of commitment, is that which underlies and underwrtites 
all these things—the ground of being, that which is One. 
I know that describing the world in terms of the “One” 
or the'ground of being” may sound mysterious, if not out- 
right mystical. But the point is not difficult to understand, 
and neither problematic nor spooky. An easier way to say 
iç, which I have used in this book many times already, is 
to say that the third level of commitment must be to that 
which we register—to that which we take to be a table, in 
the case of a table; to that which we register as a person, in 


9. An adeguate philosophical analysis of these things would reguire 
consideration of traditional categories of content, sense, reference, etc, 
But the distinction being developed here—between a patch of the 
world registered as a chair and that which is registered as a chair— 
does not normally arise, because the referent of the singular term “the 
chair” is taken to be the object that is the chair, on an assumption that 
the object exists and has identity independent of its exemplification 
of the chair property—an assumption not available in the metaphysi- 
cal view I am defending. 


the case of a person; and so on.“ Not (first stage) to the 
representation "table" or “person,' and not (second stage) to 
the table as registered as a table or to the person as registered 
as a person (since that would not leave us in a position to 
be able to say “that isn't a person after all” —or even “what 
you are taking to be a table isnt even a coherent object”). 
Rather, only if we are committed (third stage) to that which 
we take to be a table, person, and so on, will we be in a posi- 
tion to hold the registration to account. 

In my teaching, I introduce the acronym “TW” explain- 
ing it fırst as shorthand for “the world” in the sense that I 
am using it here—the “ground of being," the One, and the 
like, But as the conversation progresses it proves helpful to 
shift the connotation of the acronym (if not its meaning), 
and say that “TW” instead stands for “that which”—tche 
“that which” toward which commitment, registration, etc., 
must be oriented. 

Deference must be deference to that which we register, 
inhabit, care about, live in, and so on. Registration must 
be of that which we take to be a table, a person, or any- 
thing else. Holding registrations and registration schemes 
accountable to the that which is what it is to hold them ac- 
countable at all, 


10. “Ihat which” refers to that which plays the role of bare particulars 
in philosophical substance theories, without projecting object iden- 
tity into the world as an ontological given. 


13 — Conclusion 


Using ML and other computational technigues, including 
many from first-wave Al, we are building Al systems of 
unsurpassed reckoning power. Ihe juggernaut will only ac- 
celerate. In many realms these systems will outstrip us in 
reckoning prowess, if they do not do so already. For good 
or ill —usualiy for good, we can hope, but often for ill, we 
can also be sure— we will increasingiy delegate tasks and 
projects to them. Mining vast troves of data, and employ- 
ing computational power beyond imagining, they will in- 
creasingİy dominate the substrate and infrastructure of life 
on the planet. 

What I do not see, however, is anything on the hori- 
zon—in scientific or technological or even intellectual 
imagination—that suggests that we are about to construct, 
or indeed have any ideas as to how to construct, or are even 
thinking about constructing, systems capable of fuli-scale 
judgment: 


1. Systems existentially committed to the world they 
register, represent, and think about 

2. Systems that will go to bat for the truth, reject what 
is false, balk at what is impossible—and know the 
difference 

3. Systems not only in and of the world, but for which 
there is a world—a world that worlds, in the sense of 
constituting that to which all is ultimately accountable 


4. Systems that know that the world that hosts them, 
the entities they reason about, and all of humanity 
and community as well, must be treated with defer- 
ence, humility, and compassion 


It is this kind of jadgment, I believe—a seamless integra- 
tion of passion, dispassion, and compassion—that ulti- 
mately underwrites what matters, not just about the hu- 
man, but about the sacred, the beautiful, and the humane, 

It is this kind of judgment, I am arguing, that must be 
the aim of any project wanting to construct 'artificial gen- 
eral intelligence.” I do not believe it is inherently beyond the 
reach of synthetic construction. But it is also not an incre- 
mental advance beyond first- or second-wave Al—beyond 
the systems we have devised to date. There are profound 
differences between judgment and reckoning— especially 
the sorts of reckoning we at present have any capacity to 
construct. While predictions are a fools errand, I cannot 
see our synthesizing full scale jadgment, if indeed we are 
ever capable of doing so, in anything that anyone could call 
the short term. Even minimal progress in that direction 
will reguire strategies wholly unlike any that have been 
pursued in first- and second-wave Al. 


But we are learning. Second-wave Al has brought to our 
attention the inadeguacy of the formal ontological as- 
sumptions that underlay GOFAIJ, From its successes, and 
building on the insights of a wide diversity of other fields, 
we should have renewed respect for three interlocking and 
inexorable facts: 


ı. Ihe world is surpassingiy richa—far transcend- 
ing any ability to capture it in formal symbols, or 
for that matter in any discretely conceptualized 
structure, 

2. All (especially conceptual) registration is inevitably 
skewed, partial, and interest-relative. 

3. By exploiting yet also transcending the limitations 
of registration, genuine intelligence is committed to 
and directed toward the one and only world. 


Ihe combination of the three implies that any system that 
steps from one registration scheme to another, or that 
deals with (or is used in) different circumstances, or that 
attempts to integrate information gleaned from different 
projects, must ground its deliberations in full-scale judg- 
ment at every step of the inferential chain, in order to ensure 
that its representations never take leave of accountability to 
the world. These are strong but sobering concdusions— but 
they are straightforward conseguences of how the world is. 

Where does that leave us? We should be humbled by 
GOFAİs inadeguacy, given the depth of the insights on 
which it was based. We should be cautious about the suc- 
cesses of second-wave Al, mindful of its limitations and 
restrictions, But mostly we should stand in awe of the ca- 
pacity of the human mind, and of the achievements of hu- 
man culture, in having developed registrational strategies, 
governing norms, ontological commitments, and epistemic 
practices that allow us to comprehend and go to bat for the 
world as world. 
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